dagster dev or the Dagster UI would display an error when loading jobs created with op or asset selections.@multi_asset s are in use has been drastically improved.the_asset.with_attributes is called without providing a group name. Previously, the existing group name was erroneously dropped. Thanks @ion-elgreco!dbt source freshness.MetadataValue in Essentials course. Thanks @stianthaulow!tag_keys argument on DagsterInstance.get_run_tagsis no longer optional. This has been done to remove an easy way of accidentally executing an extremely expensive database operation.AssetSelection.dagster-snowflake now requires snowflake-connector-python>=3.4.0@sling_assets accepts an optional name parameter for the underlying opdagster-openai library is now available.DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!ConfigurableResource.SlingConnectionResource where a blank keyword argument would be emitted as an environment variabledbt source freshness would cause an error.DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.SlingConnectionResource that raised an error when connecting to a database.graph_multi_assets with check_specs now support subsetting.run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.@multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.@sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.QueuedRunCoordinator.dagster-polars has been added as an integration. Thanks @danielgafni!@dbt_assets now supports loading projects with semantic models.@dbt_assets now supports loading projects with model versions.get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!@asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!ObserveResults with no data_version.FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.k8s_job_executor.context.log.exception in a sensor or schedule did not properly capture exception information.@observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.dagster dev command that can be used to run both Dagit and the Dagster daemon in the same process during local development. See the new Running Dagster Locally guide in the docs for more information.dagster package on M1 and M2 Macs that avoids installation errors when building the grpcio package. See the Installing Dagster guide in the docs for more information.create_repository_using_definitions_args has been added for users to backport their repository definitions to the new Definitions APIdagster.yaml file:nux: enabled: false
grpcio pin in Dagster to <1.48.1 has been restored for Python versions 3.10 and 3.11, due to upstream issues in the grpcio package causing hangs in Dagster.load_assets_from_dbt* is now configurable, allowing you to pass in specific parameters to the underlying dbt command (such as --full-refresh). To do so, supply config of the form {"ops": {"run_dbt_<abcde>": {"config": {"full_refresh": True}}}} when defining your asset job, or in Dagit.build_asset_reconciliation_sensor will now only materialize partitions from the past day. Previously, if a new asset was added with a large number of missing partitions, a run for each of the missing partitions would be launched immediately.build_asset_reconciliation_sensor, which should help significantly speed up sensor evaluation in cases where there is a large number of assets or partitions.selection argument of define_asset_job now accepts lists of AssetKeys or AssetsDefinitions.RunRequest now takes a stale_assets_only flag that filters the full set of assets that would be materialized by a job to stale assets only. This can be used in schedules and sensors.dagit command and the default port 3000 is already in use.grpcio pin in Dagster to <1.48.1 has been removed for Python versions 3.10 and 3.11. Python 3.7, 3.8, and 3.9 are still pinned to <1.48.1 due to a bug in the grpc library that is causing the process to sometimes hang.stdout or stderr tab. The next time you view compute logs, you will see that tab first by default.executor and loggers arguments on Definitions are no longer experimental.json_log_format is set to False when using the dbt_cli_resource, logs will be emitted at the appropriate log level in some situations. Previously, all logs would be emitted at the INFO level.dagster-azure / dagster-aws / dagster-gcp)build_asset_reconciliation_sensor with asset graphs that contained references to source assets without the associated SourceAsset objects (which may happen when using load_assets_from_dbt_*).DbtManifestAssetSelection which could result in KeyErrors when selecting from projects with sources defined./instance and /workspace path prefixes were removed in previous version, but redirects were left in place. These redirects have now been removed.StaticPartitionMapping enables explicitly defining the dependencies between partitions in two StaticPartitionsDefinitions. Thanks Alexander VR!tag_concurrency_limits key that allows you to specify limits on the number of ops with certain tags that can be executing at once within a single run. See the docs for more information.ExecuteInProcessResult, the type returned by materialize, materialize_to_memory, and execute_in_process, now has an asset_value method that allows you to fetch output values by asset key.AssetIns can now accept Nothing for their dagster_type, which allows omitting the input from the parameters of the @asset- or @multi_asset- decorated function. This is useful when you want to specify a partition mapping or metadata for a non-managed input.start_offset and end_offset arguments of TimeWindowPartitionMapping now work across TimeWindowPartitionsDefinitions with different start dates and times.add_output_metadata is called multiple times within an op, asset, or IO manager handle_output, the values will now be merged, instead of later dictionaries overwriting earlier ones.materialize and materialize_to_memory now both accept a tags argument.SingleDimensionDependencyMapping, a PartitionMapping object that defines a correspondence between an upstream single-dimensional partitions definition and a downstream MultiPartitionsDefinition.RUN_DEQUEUED event has been removed from the event log, since it was duplicative with the RUN_STARTING event.raise Exception() from e syntax.runK8sConfig key in the k8sRunLauncher section. See the docs for more information.securityContext can now be set in the k8sRunLauncher section of the Dagster Helm chart.EcsRunLauncher can now be configured with cpu and memory resources for each launched job. Previously, individual jobs needed to be tagged with CPU and memory resources. See the docs for more information.S3ComputeLogManager now takes in an argument upload_extra_args which are passed through as the ExtraArgs parameter to the file upload call.make_dagster_definitions_from_airflow_dags_path and make_dagster_definitions_from_airflow_dag_bag which are passed through as the ExtraArgs parameter to the file upload call.ExperimentalWarnings related to LogicalVersions to appear even when version-based staleness was not in use.load_assets_from_modules and load_assets_from_package_module utilities now will also load cacheable assets from the specified modules.dequeue_num_workers config setting on QueuedRunCoordinatoris now respected.databricks_pyspark_step_launcher will now cancel the relevant databricks job if the Dagster step execution is interrupted.databricks_pyspark_step_launcher could exit with an unhelpful error after receiving an HTTPError from databricks with an empty message. This has been fixed.execute_queries or execute_query on a snowflake_resource would raise an error unless the parameters argument was explicitly set.EcsRunLauncher when launching many runs in parallel. Previously, each run risked hitting a ClientError in AWS for registering too many concurrent changes to the same task definition family. Now, the EcsRunLauncher recovers gracefully from this error by retrying it with backoff.make_dagster_definitions_from_airflow_dags_path and make_dagster_definitions_from_airflow_dag_bag for creating Dagster definitions from a given airflow Dag file path or DagBagUPathIOManager, thanks @danielgafni!FakeS3Session now includes additional functions and improvements to align with the boto3 S3 client API, thanks @asharov!BranchingIOManager to model use case where you wish to read upstream assets from production environments and write them into a development environment.create_repository_using_definitions_args to allow for the creation of named repositories.DbtManifestAssetSelection, which allows you to define selections of assets loaded from a dbt manifest using dbt selection syntax (e.g. tag:foo,path:marts/finance).@repository, replacing them with Definitions.Definitions is no longer marked as experimental and is the preferred API over @repository for new users of Dagster. Examples, tutorials, and documentation have largely ported to this new API. No migration is needed. Please see GitHub discussion for more details./locations.TimeWindowPartitionMapping now accepts start_offset and end_offset arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets.dagit can now accept multiple arguments for the -m and -f flags. For each argument a new code location is loaded.build_schedule_from_partitioned_job now execute more performantly - in constant time, rather than linear in the number of partitions.QueuedRunCoordinator now supports options dequeue_use_threads and dequeue_num_workers options to enable concurrent run dequeue operations for greater throughput.load_assets_from_dbt_project, load_assets_from_dbt_manifest, and load_assets_from_dbt_cloud_job now support applying freshness policies to loaded nodes. To do so, you can apply dagster_freshness_policy config directly in your dbt project, i.e. config(dagster_freshness_policy={"maximum_lag_minutes": 60}) would result in the corresponding asset being assigned a FreshnessPolicy(maximum_lag_minutes=60).DAGSTER_RUN_JOB_NAME environment variable is now set in containerized environments spun up by our run launchers and executor.make_dagster_repo_from_airflow_dags_path ,make_dagster_job_from_airflow_dag and make_dagster_repo_from_airflow_dag_bag have a new connections parameter which allows for configuring the airflow connections used by migrated dags.Fixed a bug where the log property was not available on the RunStatusSensorContext context object provided for run status sensors for sensor logging.
Fixed a bug where the re-execute button on runs of asset jobs would incorrectly show warning icon, indicating that the pipeline code may have changed since you last ran it.
Fixed an issue which would cause metadata supplied to graph-backed assets to not be viewable in the UI.
Fixed an issue where schedules often took up to 5 seconds to start after their tick time.
Fixed an issue where Dagster failed to load a dagster.yaml file that specified the folder to use for sqlite storage in the dagster.yaml file using an environment variable.
Fixed an issue which would cause the k8s/docker executors to unnecessarily reload CacheableAssetsDefinitions (such as those created when using load_assets_from_dbt_cloud_job) on each step execution.
[dagster-airbyte] Fixed an issue where Python-defined Airbyte sources and destinations were occasionally recreated unnecessarily.
Fixed an issue with build_asset_reconciliation_sensor that would cause it to ignore in-progress runs in some cases.
Fixed a bug where GQL errors would be thrown in the asset explorer when a previously materialized asset had its dependencies changed.
[dagster-airbyte] Fixed an error when generating assets for normalization table for connections with non-object streams.
[dagster-dbt] Fixed an error where dbt Cloud jobs with dbt run and dbt run-operation were incorrectly validated.
[dagster-airflow] use_ephemeral_airflow_db now works when running within a PEX deployment artifact.
DefinitionsDefinitions. Any content not ported to Definitions in this release is in the process of being updated.