Skip to content

Commit 5af9cc3

Browse files
ktfktf
committed
DPL: add new and improved query builder
The new `o2::framework::select()` helper can be used to construct a std::vector<InputSpec> in a much more compact way. Inputs{InputSpec{"x", "TST", "A", 0}, InputSpec{"y", "TST", "B", 0}} can now be expressed with: select("x:TST/A/0;y:TST/B/0")
1 parent e57f7e1 commit 5af9cc3

11 files changed

Lines changed: 495 additions & 24 deletions

‎Framework/Core/README.md‎

Lines changed: 12 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -83,33 +83,23 @@ struct DataProcessorSpec {
8383
};
8484
```
8585
86-
In the above both `InputSpec` and `OutputSpec` are like:
86+
the inputs field represents a set of subscriptions to some kind of input data we would like to process. As per O2 Data Model
87+
there is 3 major properties O2 attaches to a message: its origin, a description and a generic subspecification.
8788
88-
```cpp
89-
struct InputSpec {
90-
std::string binding;
91-
o2::Headers::DataDescription description;
92-
o2::Headers::DataOrigin origin;
93-
o2::Headers::SubSpecificationType subSpec;
94-
enum Lifetime lifetime;
95-
...
96-
};
97-
```
89+
So:
9890
99-
where description, origin and subSpec match the O2 Data Model definition. For the moment we will consider this a one to one mapping with the `o2::Headers::DataHeader` ones. In principle one could think of a one-to-many relationship (e.g. give me all the clusters, regardless of their provenance) and the processing layer could automatically aggregate those in a unique view. This is also the semantic difference between `InputSpec` and `OutputSpec`: the former is to express data that matches a given query (which must be exact at the moment) the latter is to describe in all details and without any doubt the kind of the produced outputs.
91+
{InputSpec{"clusters", "TPC", "CLUSTERS", 0}}
10092
101-
The `lifetime` property:
93+
really means "subscribe in input to the messages which have origin "TPC", contain objects of kind "CLUSTERS" and have a generic
94+
subspecification 0 (e.g. to index the sector). In your code you will be able to retrieve these kind of messages using the "clusters"
95+
label (see later the description of the InputRecord API).An InputSpec is effectively a three dimensional selection on space defined by all the possible origins, descriptions and subspecifications.
10296
103-
```cpp
104-
enum struct Lifetime {
105-
Timeframe,
106-
Condition,
107-
QA,
108-
Transient
109-
};
110-
```
97+
Notice that the method `o2::framework::select` can be used to provide the query
98+
in a more compact (and flexible way). Using such an helper the above becomes:
99+
100+
select("clusters:TPC/CLUSTERS/0")
111101
112-
will be used to distinguish if the associated payload should be considered payload data, and therefore be processed only once, or alignment / conditions data, and therefore it would be considered valid until a new copy is made available to the device.
102+
Similarly the `OutputSpec` is a description of what the DataProcessor will produce, again in terms of (origin, description, subspecification) properties.
113103
114104
The `configParams` vector would be used to specify which configuration options the data processing being described requires:
115105

‎Framework/Core/include/Framework/DataDescriptorMatcher.h‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
#include <array>
1919
#include <cstdint>
20+
#include <iosfwd>
2021
#include <string>
2122
#include <variant>
2223
#include <vector>
@@ -162,6 +163,16 @@ class ValueHolder
162163
return false;
163164
}
164165

166+
friend std::ostream& operator<<(std::ostream& os, ValueHolder<T> const& holder)
167+
{
168+
if (auto value = std::get_if<T>(&holder.mValue)) {
169+
os << *value;
170+
} else if (auto context = std::get_if<ContextRef>(&holder.mValue)) {
171+
os << "$" << context->index;
172+
}
173+
return os;
174+
}
175+
165176
protected:
166177
std::variant<T, ContextRef> mValue;
167178
};
@@ -400,6 +411,9 @@ class DataDescriptorMatcher
400411

401412
bool operator==(DataDescriptorMatcher const& other) const;
402413

414+
friend std::ostream& operator<<(std::ostream& os, DataDescriptorMatcher const& matcher);
415+
friend std::ostream& operator<<(std::ostream& os, Op const& matcher);
416+
403417
Node const& getLeft() const { return mLeft; };
404418
Node const& getRight() const { return mRight; };
405419
Op getOp() const { return mOp; };

‎Framework/Core/include/Framework/DataDescriptorQueryBuilder.h‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
#ifndef o2_framework_DataDescriptorQueryBuilder_H_INCLUDED
1111
#define o2_framework_DataDescriptorQueryBuilder_H_INCLUDED
1212

13+
#include "Framework/InputSpec.h"
14+
1315
#include <string>
1416
#include <vector>
1517
#include <memory>
@@ -38,6 +40,20 @@ struct DataDescriptorQuery {
3840

3941
/// Various utilities to manipulate InputSpecs
4042
struct DataDescriptorQueryBuilder {
43+
/// Creates an inputspec from a configuration @a config string with the
44+
/// following grammar.
45+
///
46+
/// string := [a-zA-Z0-9_*]*
47+
/// binding := string
48+
/// origin := string
49+
/// description := string
50+
/// subspec := [0-9]*
51+
/// spec := binding:origin/description/subspec
52+
/// config := spec;spec;...
53+
///
54+
/// Example for config: x:TPC/CLUSTER/0;y:ITS/TRACKS/1
55+
static std::vector<InputSpec> parse(const char* s = "");
56+
4157
/// Creates an inputspec from a configuration @a config string with the
4258
/// following grammar.
4359
///

‎Framework/Core/include/Framework/WorkflowSpec.h‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,17 @@ Inputs mergeInputs(Inputs inputs,
6262
/// you.
6363
DataProcessorSpec timePipeline(DataProcessorSpec original,
6464
size_t count);
65+
66+
/// The purpose of this helper is to create a query on the data via a properly formatted
67+
/// @a matcher string which describes data in terms of the O2 Data Model descriptor.
68+
///
69+
/// The syntax of the string is the following:
70+
///
71+
/// binding:origin/description/subSpecification%timemodule;...
72+
///
73+
/// Each ; delimits an InputSpec.
74+
std::vector<InputSpec> select(char const* matcher = "");
75+
6576
} // namespace framework
6677
} // namespace o2
6778

‎Framework/Core/src/DataDescriptorMatcher.cxx‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
// or submit itself to any jurisdiction.
1010

1111
#include "Framework/DataDescriptorMatcher.h"
12+
#include "Framework/DataProcessingHeader.h"
1213
#include <iostream>
1314

1415
namespace o2
@@ -78,8 +79,11 @@ bool DataDescriptorMatcher::match(ConcreteDataMatcher const& matcher, VariableCo
7879
dh.dataOrigin = matcher.origin;
7980
dh.dataDescription = matcher.description;
8081
dh.subSpecification = matcher.subSpec;
82+
DataProcessingHeader dph;
83+
dph.startTime = 0;
84+
header::Stack s{ dh, dph };
8185

82-
return this->match(reinterpret_cast<char const*>(&dh), context);
86+
return this->match(reinterpret_cast<char const*>(s.data()), context);
8387
}
8488

8589
bool DataDescriptorMatcher::match(header::DataHeader const& header, VariableContext& context) const
@@ -312,6 +316,50 @@ bool DataDescriptorMatcher::operator==(DataDescriptorMatcher const& other) const
312316
return false;
313317
}
314318

319+
std::ostream& operator<<(std::ostream& os, DataDescriptorMatcher const& matcher)
320+
{
321+
auto printer = [&os](decltype(&matcher.mLeft) v) -> void {
322+
if (auto left = std::get_if<std::unique_ptr<DataDescriptorMatcher>>(v)) {
323+
os << **left;
324+
} else if (auto originMatcher = std::get_if<OriginValueMatcher>(v)) {
325+
os << "origin:" << *originMatcher;
326+
} else if (auto descriptionMatcher = std::get_if<DescriptionValueMatcher>(v)) {
327+
os << "description:" << *descriptionMatcher;
328+
} else if (auto subSpecMatcher = std::get_if<SubSpecificationTypeValueMatcher>(v)) {
329+
os << "subSpec:" << *subSpecMatcher;
330+
} else if (auto startTimeMatcher = std::get_if<StartTimeValueMatcher>(v)) {
331+
os << "startTime:" << *startTimeMatcher;
332+
}
333+
};
334+
335+
os << "(" << matcher.mOp << " ";
336+
printer(&matcher.mLeft);
337+
os << " ";
338+
printer(&matcher.mRight);
339+
os << ")";
340+
341+
return os;
342+
}
343+
344+
std::ostream& operator<<(std::ostream& os, DataDescriptorMatcher::Op const& op)
345+
{
346+
switch (op) {
347+
case DataDescriptorMatcher::Op::And:
348+
os << "and";
349+
break;
350+
case DataDescriptorMatcher::Op::Or:
351+
os << "or";
352+
break;
353+
case DataDescriptorMatcher::Op::Just:
354+
os << "just";
355+
break;
356+
case DataDescriptorMatcher::Op::Xor:
357+
os << "xor";
358+
break;
359+
}
360+
return os;
361+
}
362+
315363
} // namespace data_matcher
316364
} // namespace framework
317365
} // namespace o2

0 commit comments

Comments
 (0)