Skip to content

Commit 22e6503

Browse files
simonbaslerobertroeser
simonbasle
authored and
robertroeser
committed
Well known mime types and Composite metadata extension (rsocket#635)
* Add encodeUnsignedMedium to NumberUtils Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add WellKnownMimeType enum with String/ID conversions Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add a flyweight capable of encoding Composite Metadata The flyweight also exposes a method to decode Composite Metadata to a Map. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Move flyweight into metadata package, change decodeToMap to decodeNext Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add CompositeMetadata interface with factory methods to decode/encode The API is intended as a readonly way of listing the composite metadata mime-buffer pairs, as well as accessing metadata pairs by mime type or index, including the case where a mime type is associated with several metadata entries (`List<Entry> getAll(String)`)... Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add first test for CompositeMetadata (decode smoke test) Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Expose incremental decode method that decodes a single metadata entry Signed-off-by: Simon Baslé <sbasle@pivotal.io> * WellKnownMimeType parsing now avoids exceptions (except null arguments) Instead has 2 special enums. int parsing allows detection of potentially legit type that is still unknown to the current implementation, which could help making it more future-proof. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * rework encoding / decoding API around Entry, accommodate 3 mime case Encoding becomes a mirror of decoding, purely based on Entry. Accommodate 3 flavors of mime type decoding: - compressed well known with id matching an enum - uncompressed custom string - compressed but known as reserved for future use (future proofness) The later can be decoded into a special Entry that will allow re encoding it and transmitting it to other nodes as it was. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * No need for a CompositeMetadata interface, single concrete implem Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Expose the CompositeMetadataFlyweight for low-level, simplify API - the flyweight deals with low-level encoding/decoding with minimal garbage and only uses exceptions in the encoding path -- encoding a non-ASCII custom mime type OR -- encoding an out of range byte - the CompositeMetadata is an _option_ for a higher level Iterable-based API, but which instantiates new types Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Remove CompositeMetadata, make Entry child of flyweight Entry is now the only high-level abstraction, will need a way to decode into a List<Entry>. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add javadoc for CompositeMetadataFlyweight methods Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Switch to a single Entry implementation, remove isPassthrough Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Rename Entry#decodeEntry to #decode, add #decodeAll Signed-off-by: Simon Baslé <sbasle@pivotal.io> * apply Google-style formatting Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Use precomputed array for WellKnownMimeType.fromId Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Use precomputed Map<String,WellKnownMimeType> for fromMimeType + jmh benchmark to validate perf increase Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Several custom mime metadata header encoding improvements: - avoid the use of a CompositeByteBuf - length can be safely predicted due to requirement that custom mime type be ASCII only - still using UTF8 writing and ByteBufUtil.isText(ASCII) to detect non-ascii text as: - writeAscii would not reject non-ascii chars but replace with `?` - ISO-8859 chars (eg. Latin-1) could be considered valid when only checking the number of written bytes - use of ByteBufUtil.writeUtf8 to simplify encoding the mime into the preallocated buffer - release the buffer if there is an encoding exception Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Use Unpooled buffers in tests Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Remove Entry, replace with CompositeMetadata Iterator-like abstraction The CompositeMetadata wraps a ByteBuf and exposes a hasNext/decodeNext API (reminiscent of the Iterator API) as a facade over the flyweight's methods. Each round decodeNext() is correctly invoked, the CompositeMetadata state is updated and the getters can be used to retrieve decoded entry components. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Add an encode flyweight method that attempts compressing String This is a qol method that will first check if the String passed can be mapped to a WellKnownMimeType (and thus compressed to a 1 byte representation). Signed-off-by: Simon Baslé <sbasle@pivotal.io> * Avoid moving full buffer's readerIndex when slicing entries This implies that the index from which to decode and slice the next entry MUST be provided by the user. To that end, we provide a method to compute the next entry's index, given the pair of ByteBuf slices returned by decodeMimeAndContentBuffers. Also renamed that method to decodeMimeAndContentBufferSlices, to make it more obvious that it produces slices (readerIndex impact and slices nature are mentioned in the javadoc). CompositeMetadata now keeps the nextEntryIndex as state and uses that for hasNext / decodeNext. Signed-off-by: Simon Baslé <sbasle@pivotal.io> * CompositeMetadata as Iterable Previously, the implementation of CompositeMetadata resulted in a mutable type that didn't fit in with either imperative (for) or reactive (fromIterable) iteration strategies. This change updates the type to implement Iterable, and return an Iterator that lazily traverses the ByteBuf. There are also some improvements to the Flyweight in order to support this iteration style. Signed-off-by: Ben Hale <bhale@pivotal.io> * Apply googleFormat Signed-off-by: Simon Baslé <sbasle@pivotal.io>
1 parent 8b8dba1 commit 22e6503

9 files changed

Lines changed: 1663 additions & 0 deletions

File tree

‎rsocket-core/src/jmh/java/io/rsocket/metadata/WellKnownMimeTypePerf.java‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package io.rsocket.metadata;
2+
3+
import org.openjdk.jmh.annotations.*;
4+
import org.openjdk.jmh.infra.Blackhole;
5+
6+
@BenchmarkMode(Mode.Throughput)
7+
@Fork(value = 1)
8+
@Warmup(iterations = 10)
9+
@Measurement(iterations = 10)
10+
@State(Scope.Thread)
11+
public class WellKnownMimeTypePerf {
12+
13+
// this is the old values() looping implementation of fromIdentifier
14+
private WellKnownMimeType fromIdValuesLoop(int id) {
15+
if (id < 0 || id > 127) {
16+
return WellKnownMimeType.UNPARSEABLE_MIME_TYPE;
17+
}
18+
for (WellKnownMimeType value : WellKnownMimeType.values()) {
19+
if (value.getIdentifier() == id) {
20+
return value;
21+
}
22+
}
23+
return WellKnownMimeType.UNKNOWN_RESERVED_MIME_TYPE;
24+
}
25+
26+
// this is the core of the old values() looping implementation of fromString
27+
private WellKnownMimeType fromStringValuesLoop(String mimeType) {
28+
for (WellKnownMimeType value : WellKnownMimeType.values()) {
29+
if (mimeType.equals(value.getString())) {
30+
return value;
31+
}
32+
}
33+
return WellKnownMimeType.UNPARSEABLE_MIME_TYPE;
34+
}
35+
36+
@Benchmark
37+
public void fromIdArrayLookup(final Blackhole bh) {
38+
// negative lookup
39+
bh.consume(WellKnownMimeType.fromIdentifier(-10));
40+
bh.consume(WellKnownMimeType.fromIdentifier(-1));
41+
// too large lookup
42+
bh.consume(WellKnownMimeType.fromIdentifier(129));
43+
// first lookup
44+
bh.consume(WellKnownMimeType.fromIdentifier(0));
45+
// middle lookup
46+
bh.consume(WellKnownMimeType.fromIdentifier(37));
47+
// reserved lookup
48+
bh.consume(WellKnownMimeType.fromIdentifier(63));
49+
// last lookup
50+
bh.consume(WellKnownMimeType.fromIdentifier(127));
51+
}
52+
53+
@Benchmark
54+
public void fromIdValuesLoopLookup(final Blackhole bh) {
55+
// negative lookup
56+
bh.consume(fromIdValuesLoop(-10));
57+
bh.consume(fromIdValuesLoop(-1));
58+
// too large lookup
59+
bh.consume(fromIdValuesLoop(129));
60+
// first lookup
61+
bh.consume(fromIdValuesLoop(0));
62+
// middle lookup
63+
bh.consume(fromIdValuesLoop(37));
64+
// reserved lookup
65+
bh.consume(fromIdValuesLoop(63));
66+
// last lookup
67+
bh.consume(fromIdValuesLoop(127));
68+
}
69+
70+
@Benchmark
71+
public void fromStringMapLookup(final Blackhole bh) {
72+
// unknown lookup
73+
bh.consume(WellKnownMimeType.fromString("foo/bar"));
74+
// first lookup
75+
bh.consume(WellKnownMimeType.fromString(WellKnownMimeType.APPLICATION_AVRO.getString()));
76+
// middle lookup
77+
bh.consume(WellKnownMimeType.fromString(WellKnownMimeType.VIDEO_VP8.getString()));
78+
// last lookup
79+
bh.consume(
80+
WellKnownMimeType.fromString(
81+
WellKnownMimeType.MESSAGE_RSOCKET_COMPOSITE_METADATA.getString()));
82+
}
83+
84+
@Benchmark
85+
public void fromStringValuesLoopLookup(final Blackhole bh) {
86+
// unknown lookup
87+
bh.consume(fromStringValuesLoop("foo/bar"));
88+
// first lookup
89+
bh.consume(fromStringValuesLoop(WellKnownMimeType.APPLICATION_AVRO.getString()));
90+
// middle lookup
91+
bh.consume(fromStringValuesLoop(WellKnownMimeType.VIDEO_VP8.getString()));
92+
// last lookup
93+
bh.consume(
94+
fromStringValuesLoop(WellKnownMimeType.MESSAGE_RSOCKET_COMPOSITE_METADATA.getString()));
95+
}
96+
}

‎rsocket-core/src/main/java/io/rsocket/metadata/CompositeMetadata.java‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
/*
2+
* Copyright 2015-2018 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package io.rsocket.metadata;
18+
19+
import static io.rsocket.metadata.CompositeMetadataFlyweight.computeNextEntryIndex;
20+
import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeAndContentBuffersSlices;
21+
import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeIdFromMimeBuffer;
22+
import static io.rsocket.metadata.CompositeMetadataFlyweight.decodeMimeTypeFromMimeBuffer;
23+
import static io.rsocket.metadata.CompositeMetadataFlyweight.hasEntry;
24+
import static io.rsocket.metadata.CompositeMetadataFlyweight.isWellKnownMimeType;
25+
26+
import io.netty.buffer.ByteBuf;
27+
import io.netty.buffer.ByteBufAllocator;
28+
import io.netty.buffer.CompositeByteBuf;
29+
import io.rsocket.metadata.CompositeMetadata.Entry;
30+
import java.util.Iterator;
31+
import reactor.util.annotation.Nullable;
32+
33+
/**
34+
* An {@link Iterable} wrapper around a {@link ByteBuf} that exposes metadata entry information at
35+
* each decoding step. This is only possible on frame types used to initiate interactions, if the
36+
* SETUP metadata mime type was {@link WellKnownMimeType#MESSAGE_RSOCKET_COMPOSITE_METADATA}.
37+
*
38+
* <p>This allows efficient incremental decoding of the entries (without moving the source's {@link
39+
* io.netty.buffer.ByteBuf#readerIndex()}). The buffer is assumed to contain just enough bytes to
40+
* represent one or more entries (mime type compressed or not). The decoding stops when the buffer
41+
* reaches 0 readable bytes, and fails if it contains bytes but not enough to correctly decode an
42+
* entry.
43+
*
44+
* <p>A note on future-proofness: it is possible to come across a compressed mime type that this
45+
* implementation doesn't recognize. This is likely to be due to the use of a byte id that is merely
46+
* reserved in this implementation, but maps to a {@link WellKnownMimeType} in the implementation
47+
* that encoded the metadata. This can be detected by detecting that an entry is a {@link
48+
* ReservedMimeTypeEntry}. In this case {@link Entry#getMimeType()} will return {@code null}. The
49+
* encoded id can be retrieved using {@link ReservedMimeTypeEntry#getType()}. The byte and content
50+
* buffer should be kept around and re-encoded using {@link
51+
* CompositeMetadataFlyweight#encodeAndAddMetadata(CompositeByteBuf, ByteBufAllocator, byte,
52+
* ByteBuf)} in case passing that entry through is required.
53+
*/
54+
public final class CompositeMetadata implements Iterable<Entry> {
55+
56+
private final boolean retainSlices;
57+
58+
private final ByteBuf source;
59+
60+
public CompositeMetadata(ByteBuf source, boolean retainSlices) {
61+
this.source = source;
62+
this.retainSlices = retainSlices;
63+
}
64+
65+
@Override
66+
public Iterator<Entry> iterator() {
67+
return new Iterator<Entry>() {
68+
69+
private int entryIndex = 0;
70+
71+
@Override
72+
public boolean hasNext() {
73+
return hasEntry(CompositeMetadata.this.source, this.entryIndex);
74+
}
75+
76+
@Override
77+
public Entry next() {
78+
ByteBuf[] headerAndData =
79+
decodeMimeAndContentBuffersSlices(
80+
CompositeMetadata.this.source,
81+
this.entryIndex,
82+
CompositeMetadata.this.retainSlices);
83+
84+
ByteBuf header = headerAndData[0];
85+
ByteBuf data = headerAndData[1];
86+
87+
this.entryIndex = computeNextEntryIndex(this.entryIndex, header, data);
88+
89+
if (!isWellKnownMimeType(header)) {
90+
CharSequence typeString = decodeMimeTypeFromMimeBuffer(header);
91+
if (typeString == null) {
92+
throw new IllegalStateException("MIME type cannot be null");
93+
}
94+
95+
return new ExplicitMimeTimeEntry(data, typeString.toString());
96+
}
97+
98+
byte id = decodeMimeIdFromMimeBuffer(header);
99+
WellKnownMimeType type = WellKnownMimeType.fromIdentifier(id);
100+
101+
if (WellKnownMimeType.UNKNOWN_RESERVED_MIME_TYPE == type) {
102+
return new ReservedMimeTypeEntry(data, id);
103+
}
104+
105+
return new WellKnownMimeTypeEntry(data, type);
106+
}
107+
};
108+
}
109+
110+
/** An entry in the {@link CompositeMetadata}. */
111+
public interface Entry {
112+
113+
/**
114+
* Returns the un-decoded content of the {@link Entry}.
115+
*
116+
* @return the un-decoded content of the {@link Entry}
117+
*/
118+
ByteBuf getContent();
119+
120+
/**
121+
* Returns the MIME type of the entry, if it can be decoded.
122+
*
123+
* @return the MIME type of the entry, if it can be decoded, otherwise {@code null}.
124+
*/
125+
@Nullable
126+
String getMimeType();
127+
}
128+
129+
/** An {@link Entry} backed by an explicitly declared MIME type. */
130+
public static final class ExplicitMimeTimeEntry implements Entry {
131+
132+
private final ByteBuf content;
133+
134+
private final String type;
135+
136+
public ExplicitMimeTimeEntry(ByteBuf content, String type) {
137+
this.content = content;
138+
this.type = type;
139+
}
140+
141+
@Override
142+
public ByteBuf getContent() {
143+
return this.content;
144+
}
145+
146+
@Override
147+
public String getMimeType() {
148+
return this.type;
149+
}
150+
}
151+
152+
/**
153+
* An {@link Entry} backed by a {@link WellKnownMimeType} entry, but one that is not understood by
154+
* this implementation.
155+
*/
156+
public static final class ReservedMimeTypeEntry implements Entry {
157+
private final ByteBuf content;
158+
private final int type;
159+
160+
public ReservedMimeTypeEntry(ByteBuf content, int type) {
161+
this.content = content;
162+
this.type = type;
163+
}
164+
165+
@Override
166+
public ByteBuf getContent() {
167+
return this.content;
168+
}
169+
170+
/**
171+
* {@inheritDoc} Since this entry represents a compressed id that couldn't be decoded, this is
172+
* always {@code null}.
173+
*/
174+
@Override
175+
public String getMimeType() {
176+
return null;
177+
}
178+
179+
/**
180+
* Returns the reserved, but unknown {@link WellKnownMimeType} for this entry. Range is 0-127
181+
* (inclusive).
182+
*
183+
* @return the reserved, but unknown {@link WellKnownMimeType} for this entry
184+
*/
185+
public int getType() {
186+
return this.type;
187+
}
188+
}
189+
190+
/** An {@link Entry} backed by a {@link WellKnownMimeType}. */
191+
public static final class WellKnownMimeTypeEntry implements Entry {
192+
193+
private final ByteBuf content;
194+
private final WellKnownMimeType type;
195+
196+
public WellKnownMimeTypeEntry(ByteBuf content, WellKnownMimeType type) {
197+
this.content = content;
198+
this.type = type;
199+
}
200+
201+
@Override
202+
public ByteBuf getContent() {
203+
return this.content;
204+
}
205+
206+
@Override
207+
public String getMimeType() {
208+
return this.type.getString();
209+
}
210+
211+
/**
212+
* Returns the {@link WellKnownMimeType} for this entry.
213+
*
214+
* @return the {@link WellKnownMimeType} for this entry
215+
*/
216+
public WellKnownMimeType getType() {
217+
return this.type;
218+
}
219+
}
220+
}

0 commit comments

Comments
 (0)