ShardFilter (autotransform.filter.shard)

A base for sharding filters used by schedule runs to shard Input items.

class autotransform.filter.shard.ShardFilter(*, inverted: bool = False, num_shards: int, valid_shard: int = -1)

Bases: Filter

A base for sharding filters that checks that an Item fits the current valid shard.

num_shards

The number of shards to split the items across.

Type:

int

valid_shard

The current valid shard to use.

Type:

int

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[dict[str, FieldInfo]] = {'inverted': FieldInfo(annotation=bool, required=False, default=False), 'num_shards': FieldInfo(annotation=int, required=True), 'valid_shard': FieldInfo(annotation=int, required=False, default=-1)}

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo].

This replaces Model.__fields__ from Pydantic V1.

num_shards: int

valid_shard: int