bar

+```````````````````````````````` + + + +A partial tag need not even be completed (garbage +in, garbage out): + +```````````````````````````````` example +

*foo*

+. +

*foo*

+```````````````````````````````` + + +```````````````````````````````` example +

+foo +

+. +

+foo +

+```````````````````````````````` + + +Everything until the next blank line or end of document +gets included in the HTML block. So, in the following +example, what looks like a Markdown code block +is actually part of the HTML block, which continues until a blank +line or the end of the document is reached: + +```````````````````````````````` example +

+``` c +int x = 33; +``` +. +

+``` c +int x = 33; +``` +```````````````````````````````` + + +To start an [HTML block] with a tag that is *not* in the +list of block-level tags in (6), you must put the tag by +itself on the first line (and it must be complete): + +```````````````````````````````` example + +*bar* + +. + +*bar* + +```````````````````````````````` + + +In type 7 blocks, the [tag name] can be anything: + +```````````````````````````````` example + +*bar* + +. + +*bar* + +```````````````````````````````` + + +```````````````````````````````` example + +*bar* + +. + +*bar* + +```````````````````````````````` + + +```````````````````````````````` example + +*bar* +. + +*bar* +```````````````````````````````` + + +These rules are designed to allow us to work with tags that +can function as either block-level or inline-level tags. +The `` tag is a nice example. We can surround content with +`` tags in three different ways. In this case, we get a raw +HTML block, because the `` tag is on a line by itself: + +```````````````````````````````` example + ~~+*foo* +~~ +. + ~~+*foo* +~~ +```````````````````````````````` + + +In this case, we get a raw HTML block that just includes +the `` tag (because it ends with the following blank +line). So the contents get interpreted as CommonMark: + +```````````````````````````````` example + ~~+ +*foo* + +~~ +. + +
foo
+ +```````````````````````````````` + + +Finally, in this case, the `` tags are interpreted +as [raw HTML] *inside* the CommonMark paragraph. (Because +the tag is not on a line by itself, we get inline HTML +rather than an [HTML block].) + +```````````````````````````````` example +~~*foo*~~ +. +
~~foo~~
+```````````````````````````````` + + +HTML tags designed to contain literal content +(`pre`, `script`, `style`, `textarea`), comments, processing instructions, +and declarations are treated somewhat differently. +Instead of ending at the first blank line, these blocks +end at the first line containing a corresponding end tag. +As a result, these blocks can contain blank lines: + +A pre tag (type 1): + +```````````````````````````````` example +
+import Text.HTML.TagSoup + +main :: IO () +main = print $ parseTags tags +
+okay +. +
+import Text.HTML.TagSoup + +main :: IO () +main = print $ parseTags tags +
+
okay
+```````````````````````````````` + + +A script tag (type 1): + +```````````````````````````````` example + +okay +. + +
okay
+```````````````````````````````` + + +A textarea tag (type 1): + +```````````````````````````````` example ++ +*foo* + +_bar_ + + +. ++ +*foo* + +_bar_ + + +```````````````````````````````` + +A style tag (type 1): + +```````````````````````````````` example + +okay +. + +
okay
+```````````````````````````````` + + +If there is no matching end tag, the block will end at the +end of the document (or the enclosing [block quote][block quotes] +or [list item][list items]): + +```````````````````````````````` example + +*foo* +. + +
foo
+```````````````````````````````` + + +```````````````````````````````` example +*bar* +*baz* +. +*bar* +
baz
+```````````````````````````````` + + +Note that anything on the last line after the +end tag will be included in the [HTML block]: + +```````````````````````````````` example +1. *bar* +. +1. *bar* +```````````````````````````````` + + +A comment (type 2): + +```````````````````````````````` example + +okay +. + +
okay
+```````````````````````````````` + + + +A processing instruction (type 3): + +```````````````````````````````` example +'; + +?> +okay +. +'; + +?> +
okay
+```````````````````````````````` + + +A declaration (type 4): + +```````````````````````````````` example + +. + +```````````````````````````````` + + +CDATA (type 5): + +```````````````````````````````` example + +okay +. + +
okay
+```````````````````````````````` + + +The opening tag can be preceded by up to three spaces of indentation, but not +four: + +```````````````````````````````` example + + + +. + +
 +
+```````````````````````````````` + + +```````````````````````````````` example +
+ +
+. +
+
<div> +
+```````````````````````````````` + + +An HTML block of types 1--6 can interrupt a paragraph, and need not be +preceded by a blank line. + +```````````````````````````````` example +Foo +
+bar +
+. +
Foo
+
+bar +
+```````````````````````````````` + + +However, a following blank line is needed, except at the end of +a document, and except for blocks of types 1--5, [above][HTML +block]: + +```````````````````````````````` example +
+bar +
+*foo* +. +
+bar +
+*foo* +```````````````````````````````` + + +HTML blocks of type 7 cannot interrupt a paragraph: + +```````````````````````````````` example +Foo + +baz +. +
Foo + +baz
+```````````````````````````````` + + +This rule differs from John Gruber's original Markdown syntax +specification, which says: + +> The only restrictions are that block-level HTML elements — +> e.g. `
`, ``, `
`, `
`, etc. — must be separated from +> surrounding content by blank lines, and the start and end tags of the +> block should not be indented with spaces or tabs. + +In some ways Gruber's rule is more restrictive than the one given +here: + +- It requires that an HTML block be preceded by a blank line. +- It does not allow the start tag to be indented. +- It requires a matching end tag, which it also does not allow to + be indented. + +Most Markdown implementations (including some of Gruber's own) do not +respect all of these restrictions. + +There is one respect, however, in which Gruber's rule is more liberal +than the one given here, since it allows blank lines to occur inside +an HTML block. There are two reasons for disallowing them here. +First, it removes the need to parse balanced tags, which is +expensive and can require backtracking from the end of the document +if no matching end tag is found. Second, it provides a very simple +and flexible way of including Markdown content inside HTML tags: +simply separate the Markdown from the HTML using blank lines: + +Compare: + +```````````````````````````````` example +
+ +*Emphasized* text. + +
+. +
+
Emphasized text.
+
+```````````````````````````````` + + +```````````````````````````````` example +
+*Emphasized* text. +
+. +
+*Emphasized* text. +
+```````````````````````````````` + + +Some Markdown implementations have adopted a convention of +interpreting content inside tags as text if the open tag has +the attribute `markdown=1`. The rule given above seems a simpler and +more elegant way of achieving the same expressive power, which is also +much simpler to parse. + +The main potential drawback is that one can no longer paste HTML +blocks into Markdown documents with 100% reliability. However, +*in most cases* this will work fine, because the blank lines in +HTML are usually followed by HTML block tags. For example: + +```````````````````````````````` example +
+ + + + + + + +
+Hi +
+. + + + + +
+Hi +
+```````````````````````````````` + + +There are problems, however, if the inner tags are indented +*and* separated by spaces, as then they will be interpreted as +an indented code block: + +```````````````````````````````` example + + + + + + + + +
+ Hi +
+. + + +
<td> + Hi +</td> +
+ +
+```````````````````````````````` + + +Fortunately, blank lines are usually not necessary and can be +deleted. The exception is inside `
` tags, but as described +[above][HTML blocks], raw HTML blocks starting with `
` +*can* contain blank lines. + +## Link reference definitions + +A [link reference definition](@) +consists of a [link label], optionally preceded by up to three spaces of +indentation, followed +by a colon (`:`), optional spaces or tabs (including up to one +[line ending]), a [link destination], +optional spaces or tabs (including up to one +[line ending]), and an optional [link +title], which if it is present must be separated +from the [link destination] by spaces or tabs. +No further character may occur. + +A [link reference definition] +does not correspond to a structural element of a document. Instead, it +defines a label which can be used in [reference links] +and reference-style [images] elsewhere in the document. [Link +reference definitions] can come either before or after the links that use +them. + +```````````````````````````````` example +[foo]: /url "title" + +[foo] +. +
foo
+```````````````````````````````` + + +```````````````````````````````` example + [foo]: + /url + 'the title' + +[foo] +. +
foo
+```````````````````````````````` + + +```````````````````````````````` example +[Foo*bar\]]:my_(url) 'title (with parens)' + +[Foo*bar\]] +. +
Foo*bar]
+```````````````````````````````` + + +```````````````````````````````` example +[Foo bar]: + +'title' + +[Foo bar] +. +
Foo bar
+```````````````````````````````` + + +The title may extend over multiple lines: + +```````````````````````````````` example +[foo]: /url ' +title +line1 +line2 +' + +[foo] +. +
foo
+```````````````````````````````` + + +However, it may not contain a [blank line]: + +```````````````````````````````` example +[foo]: /url 'title + +with blank line' + +[foo] +. +
[foo]: /url 'title
+
with blank line'
+
[foo]
+```````````````````````````````` + + +The title may be omitted: + +```````````````````````````````` example +[foo]: +/url + +[foo] +. +
foo
+```````````````````````````````` + + +The link destination may not be omitted: + +```````````````````````````````` example +[foo]: + +[foo] +. +
[foo]:
+
[foo]
+```````````````````````````````` + + However, an empty link destination may be specified using + angle brackets: + +```````````````````````````````` example +[foo]: <> + +[foo] +. +
foo
+```````````````````````````````` + +The title must be separated from the link destination by +spaces or tabs: + +```````````````````````````````` example +[foo]: (baz) + +[foo] +. +
[foo]: (baz)
+
[foo]
+```````````````````````````````` + + +Both title and destination can contain backslash escapes +and literal backslashes: + +```````````````````````````````` example +[foo]: /url\bar\*baz "foo\"bar\baz" + +[foo] +. +
foo
+```````````````````````````````` + + +A link can come before its corresponding definition: + +```````````````````````````````` example +[foo] + +[foo]: url +. +
foo
+```````````````````````````````` + + +If there are several matching definitions, the first one takes +precedence: + +```````````````````````````````` example +[foo] + +[foo]: first +[foo]: second +. +
foo
+```````````````````````````````` + + +As noted in the section on [Links], matching of labels is +case-insensitive (see [matches]). + +```````````````````````````````` example +[FOO]: /url + +[Foo] +. +
Foo
+```````````````````````````````` + + +```````````````````````````````` example +[ΑΓΩ]: /φου + +[αγω] +. +
αγω
+```````````````````````````````` + + +Whether something is a [link reference definition] is +independent of whether the link reference it defines is +used in the document. Thus, for example, the following +document contains just a link reference definition, and +no visible content: + +```````````````````````````````` example +[foo]: /url +. +```````````````````````````````` + + +Here is another one: + +```````````````````````````````` example +[ +foo +]: /url +bar +. +
bar
+```````````````````````````````` + + +This is not a link reference definition, because there are +characters other than spaces or tabs after the title: + +```````````````````````````````` example +[foo]: /url "title" ok +. +
[foo]: /url "title" ok
+```````````````````````````````` + + +This is a link reference definition, but it has no title: + +```````````````````````````````` example +[foo]: /url +"title" ok +. +
"title" ok
+```````````````````````````````` + + +This is not a link reference definition, because it is indented +four spaces: + +```````````````````````````````` example + [foo]: /url "title" + +[foo] +. +
[foo]: /url "title" +
+
[foo]
+```````````````````````````````` + + +This is not a link reference definition, because it occurs inside +a code block: + +```````````````````````````````` example +``` +[foo]: /url +``` + +[foo] +. +
[foo]: /url +
+
[foo]
+```````````````````````````````` + + +A [link reference definition] cannot interrupt a paragraph. + +```````````````````````````````` example +Foo +[bar]: /baz + +[bar] +. +
Foo +[bar]: /baz
+
[bar]
+```````````````````````````````` + + +However, it can directly follow other block elements, such as headings +and thematic breaks, and it need not be followed by a blank line. + +```````````````````````````````` example +# [Foo] +[foo]: /url +> bar +. +
Foo
+
+
bar
+
+```````````````````````````````` + +```````````````````````````````` example +[foo]: /url +bar +=== +[foo] +. +
bar
+
foo
+```````````````````````````````` + +```````````````````````````````` example +[foo]: /url +=== +[foo] +. +
=== +foo
+```````````````````````````````` + + +Several [link reference definitions] +can occur one after another, without intervening blank lines. + +```````````````````````````````` example +[foo]: /foo-url "foo" +[bar]: /bar-url + "bar" +[baz]: /baz-url + +[foo], +[bar], +[baz] +. +
foo, +bar, +baz
+```````````````````````````````` + + +[Link reference definitions] can occur +inside block containers, like lists and block quotations. They +affect the entire document, not just the container in which they +are defined: + +```````````````````````````````` example +[foo] + +> [foo]: /url +. +
foo
+
+
+```````````````````````````````` + + +## Paragraphs + +A sequence of non-blank lines that cannot be interpreted as other +kinds of blocks forms a [paragraph](@). +The contents of the paragraph are the result of parsing the +paragraph's raw content as inlines. The paragraph's raw content +is formed by concatenating the lines and removing initial and final +spaces or tabs. + +A simple example with two paragraphs: + +```````````````````````````````` example +aaa + +bbb +. +
aaa
+
bbb
+```````````````````````````````` + + +Paragraphs can contain multiple lines, but no blank lines: + +```````````````````````````````` example +aaa +bbb + +ccc +ddd +. +
aaa +bbb
+
ccc +ddd
+```````````````````````````````` + + +Multiple blank lines between paragraphs have no effect: + +```````````````````````````````` example +aaa + + +bbb +. +
aaa
+
bbb
+```````````````````````````````` + + +Leading spaces or tabs are skipped: + +```````````````````````````````` example + aaa + bbb +. +
aaa +bbb
+```````````````````````````````` + + +Lines after the first may be indented any amount, since indented +code blocks cannot interrupt paragraphs. + +```````````````````````````````` example +aaa + bbb + ccc +. +
aaa +bbb +ccc
+```````````````````````````````` + + +However, the first line may be preceded by up to three spaces of indentation. +Four spaces of indentation is too many: + +```````````````````````````````` example + aaa +bbb +. +
aaa +bbb
+```````````````````````````````` + + +```````````````````````````````` example + aaa +bbb +. +
aaa +
+
bbb
+```````````````````````````````` + + +Final spaces or tabs are stripped before inline parsing, so a paragraph +that ends with two or more spaces will not end with a [hard line +break]: + +```````````````````````````````` example +aaa +bbb +. +
aaa
+bbb
+```````````````````````````````` + + +## Blank lines + +[Blank lines] between block-level elements are ignored, +except for the role they play in determining whether a [list] +is [tight] or [loose]. + +Blank lines at the beginning and end of the document are also ignored. + +```````````````````````````````` example + + +aaa + + +# aaa + + +. +
aaa
+
aaa
+```````````````````````````````` + + + +# Container blocks + +A [container block](#container-blocks) is a block that has other +blocks as its contents. There are two basic kinds of container blocks: +[block quotes] and [list items]. +[Lists] are meta-containers for [list items]. + +We define the syntax for container blocks recursively. The general +form of the definition is: + +> If X is a sequence of blocks, then the result of +> transforming X in such-and-such a way is a container of type Y +> with these blocks as its content. + +So, we explain what counts as a block quote or list item by explaining +how these can be *generated* from their contents. This should suffice +to define the syntax, although it does not give a recipe for *parsing* +these constructions. (A recipe is provided below in the section entitled +[A parsing strategy](#appendix-a-parsing-strategy).) + +## Block quotes + +A [block quote marker](@), +optionally preceded by up to three spaces of indentation, +consists of (a) the character `>` together with a following space of +indentation, or (b) a single character `>` not followed by a space of +indentation. + +The following rules define [block quotes]: + +1. **Basic case.** If a string of lines *Ls* constitute a sequence + of blocks *Bs*, then the result of prepending a [block quote + marker] to the beginning of each line in *Ls* + is a [block quote](#block-quotes) containing *Bs*. + +2. **Laziness.** If a string of lines *Ls* constitute a [block + quote](#block-quotes) with contents *Bs*, then the result of deleting + the initial [block quote marker] from one or + more lines in which the next character other than a space or tab after the + [block quote marker] is [paragraph continuation + text] is a block quote with *Bs* as its content. + [Paragraph continuation text](@) is text + that will be parsed as part of the content of a paragraph, but does + not occur at the beginning of the paragraph. + +3. **Consecutiveness.** A document cannot contain two [block + quotes] in a row unless there is a [blank line] between them. + +Nothing else counts as a [block quote](#block-quotes). + +Here is a simple example: + +```````````````````````````````` example +> # Foo +> bar +> baz +. +
+
Foo
+
bar +baz
+
+```````````````````````````````` + + +The space or tab after the `>` characters can be omitted: + +```````````````````````````````` example +># Foo +>bar +> baz +. +
+
Foo
+
bar +baz
+
+```````````````````````````````` + + +The `>` characters can be preceded by up to three spaces of indentation: + +```````````````````````````````` example + > # Foo + > bar + > baz +. +
+
Foo
+
bar +baz
+
+```````````````````````````````` + + +Four spaces of indentation is too many: + +```````````````````````````````` example + > # Foo + > bar + > baz +. +
> # Foo +> bar +> baz +
+```````````````````````````````` + + +The Laziness clause allows us to omit the `>` before +[paragraph continuation text]: + +```````````````````````````````` example +> # Foo +> bar +baz +. +
+
Foo
+
bar +baz
+
+```````````````````````````````` + + +A block quote can contain some lazy and some non-lazy +continuation lines: + +```````````````````````````````` example +> bar +baz +> foo +. +
+
bar +baz +foo
+
+```````````````````````````````` + + +Laziness only applies to lines that would have been continuations of +paragraphs had they been prepended with [block quote markers]. +For example, the `> ` cannot be omitted in the second line of + +``` markdown +> foo +> --- +``` + +without changing the meaning: + +```````````````````````````````` example +> foo +--- +. +
+
foo
+
+
+```````````````````````````````` + + +Similarly, if we omit the `> ` in the second line of + +``` markdown +> - foo +> - bar +``` + +then the block quote ends after the first line: + +```````````````````````````````` example +> - foo +- bar +. +
+
+
foo
+
+
+
+
bar
+
+```````````````````````````````` + + +For the same reason, we can't omit the `> ` in front of +subsequent lines of an indented or fenced code block: + +```````````````````````````````` example +> foo + bar +. +
+
foo +
+
+
bar +
+```````````````````````````````` + + +```````````````````````````````` example +> ``` +foo +``` +. +
+
+
+
foo
+
+```````````````````````````````` + + +Note that in the following case, we have a [lazy +continuation line]: + +```````````````````````````````` example +> foo + - bar +. +
+
foo +- bar
+
+```````````````````````````````` + + +To see why, note that in + +```markdown +> foo +> - bar +``` + +the `- bar` is indented too far to start a list, and can't +be an indented code block because indented code blocks cannot +interrupt paragraphs, so it is [paragraph continuation text]. + +A block quote can be empty: + +```````````````````````````````` example +> +. +
+
+```````````````````````````````` + + +```````````````````````````````` example +> +> +> +. +
+
+```````````````````````````````` + + +A block quote can have initial or final blank lines: + +```````````````````````````````` example +> +> foo +> +. +
+
foo
+
+```````````````````````````````` + + +A blank line always separates block quotes: + +```````````````````````````````` example +> foo + +> bar +. +
+
foo
+
+
+
bar
+
+```````````````````````````````` + + +(Most current Markdown implementations, including John Gruber's +original `Markdown.pl`, will parse this example as a single block quote +with two paragraphs. But it seems better to allow the author to decide +whether two block quotes or one are wanted.) + +Consecutiveness means that if we put these block quotes together, +we get a single block quote: + +```````````````````````````````` example +> foo +> bar +. +
+
foo +bar
+
+```````````````````````````````` + + +To get a block quote with two paragraphs, use: + +```````````````````````````````` example +> foo +> +> bar +. +
+
foo
+
bar
+
+```````````````````````````````` + + +Block quotes can interrupt paragraphs: + +```````````````````````````````` example +foo +> bar +. +
foo
+
+
bar
+
+```````````````````````````````` + + +In general, blank lines are not needed before or after block +quotes: + +```````````````````````````````` example +> aaa +*** +> bbb +. +
+
aaa
+
+
+
+
bbb
+
+```````````````````````````````` + + +However, because of laziness, a blank line is needed between +a block quote and a following paragraph: + +```````````````````````````````` example +> bar +baz +. +
+
bar +baz
+
+```````````````````````````````` + + +```````````````````````````````` example +> bar + +baz +. +
+
bar
+
+
baz
+```````````````````````````````` + + +```````````````````````````````` example +> bar +> +baz +. +
+
bar
+
+
baz
+```````````````````````````````` + + +It is a consequence of the Laziness rule that any number +of initial `>`s may be omitted on a continuation line of a +nested block quote: + +```````````````````````````````` example +> > > foo +bar +. +
+
+
+
foo +bar
+
+
+
+```````````````````````````````` + + +```````````````````````````````` example +>>> foo +> bar +>>baz +. +
+
+
+
foo +bar +baz
+
+
+
+```````````````````````````````` + + +When including an indented code block in a block quote, +remember that the [block quote marker] includes +both the `>` and a following space of indentation. So *five spaces* are needed +after the `>`: + +```````````````````````````````` example +> code + +> not code +. +
+
code +
+
+
+
not code
+
+```````````````````````````````` + + + +## List items + +A [list marker](@) is a +[bullet list marker] or an [ordered list marker]. + +A [bullet list marker](@) +is a `-`, `+`, or `*` character. + +An [ordered list marker](@) +is a sequence of 1--9 arabic digits (`0-9`), followed by either a +`.` character or a `)` character. (The reason for the length +limit is that with 10 digits we start seeing integer overflows +in some browsers.) + +The following rules define [list items]: + +1. **Basic case.** If a sequence of lines *Ls* constitute a sequence of + blocks *Bs* starting with a character other than a space or tab, and *M* is + a list marker of width *W* followed by 1 ≤ *N* ≤ 4 spaces of indentation, + then the result of prepending *M* and the following spaces to the first line + of Ls*, and indenting subsequent lines of *Ls* by *W + N* spaces, is a + list item with *Bs* as its contents. The type of the list item + (bullet or ordered) is determined by the type of its list marker. + If the list item is ordered, then it is also assigned a start + number, based on the ordered list marker. + + Exceptions: + + 1. When the first list item in a [list] interrupts + a paragraph---that is, when it starts on a line that would + otherwise count as [paragraph continuation text]---then (a) + the lines *Ls* must not begin with a blank line, and (b) if + the list item is ordered, the start number must be 1. + 2. If any line is a [thematic break][thematic breaks] then + that line is not a list item. + +For example, let *Ls* be the lines + +```````````````````````````````` example +A paragraph +with two lines. + + indented code + +> A block quote. +. +
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+```````````````````````````````` + + +And let *M* be the marker `1.`, and *N* = 2. Then rule #1 says +that the following is an ordered list item with start number 1, +and the same contents as *Ls*: + +```````````````````````````````` example +1. A paragraph + with two lines. + + indented code + + > A block quote. +. +
+
+
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+
+
+```````````````````````````````` + + +The most important thing to notice is that the position of +the text after the list marker determines how much indentation +is needed in subsequent blocks in the list item. If the list +marker takes up two spaces of indentation, and there are three spaces between +the list marker and the next character other than a space or tab, then blocks +must be indented five spaces in order to fall under the list +item. + +Here are some examples showing how far content must be indented to be +put under the list item: + +```````````````````````````````` example +- one + + two +. +
+
one
+
+
two
+```````````````````````````````` + + +```````````````````````````````` example +- one + + two +. +
+
+
one
+
two
+
+
+```````````````````````````````` + + +```````````````````````````````` example + - one + + two +. +
+
one
+
+
two +
+```````````````````````````````` + + +```````````````````````````````` example + - one + + two +. +
+
+
one
+
two
+
+
+```````````````````````````````` + + +It is tempting to think of this in terms of columns: the continuation +blocks must be indented at least to the column of the first character other than +a space or tab after the list marker. However, that is not quite right. +The spaces of indentation after the list marker determine how much relative +indentation is needed. Which column this indentation reaches will depend on +how the list item is embedded in other constructions, as shown by +this example: + +```````````````````````````````` example + > > 1. one +>> +>> two +. +
+
+
+
+
one
+
two
+
+
+
+
+```````````````````````````````` + + +Here `two` occurs in the same column as the list marker `1.`, +but is actually contained in the list item, because there is +sufficient indentation after the last containing blockquote marker. + +The converse is also possible. In the following example, the word `two` +occurs far to the right of the initial text of the list item, `one`, but +it is not considered part of the list item, because it is not indented +far enough past the blockquote marker: + +```````````````````````````````` example +>>- one +>> + > > two +. +
+
+
+
one
+
+
two
+
+
+```````````````````````````````` + + +Note that at least one space or tab is needed between the list marker and +any following content, so these are not list items: + +```````````````````````````````` example +-one + +2.two +. +
-one
+
2.two
+```````````````````````````````` + + +A list item may contain blocks that are separated by more than +one blank line. + +```````````````````````````````` example +- foo + + + bar +. +
+
+
foo
+
bar
+
+
+```````````````````````````````` + + +A list item may contain any kind of block: + +```````````````````````````````` example +1. foo + + ``` + bar + ``` + + baz + + > bam +. +
+
+
foo
+
bar +
+
baz
+
+
bam
+
+
+
+```````````````````````````````` + + +A list item that contains an indented code block will preserve +empty lines within the code block verbatim. + +```````````````````````````````` example +- Foo + + bar + + + baz +. +
+
+
Foo
+
bar + + +baz +
+
+
+```````````````````````````````` + +Note that ordered list start numbers must be nine digits or less: + +```````````````````````````````` example +123456789. ok +. +
+
ok
+
+```````````````````````````````` + + +```````````````````````````````` example +1234567890. not ok +. +
1234567890. not ok
+```````````````````````````````` + + +A start number may begin with 0s: + +```````````````````````````````` example +0. ok +. +
+
ok
+
+```````````````````````````````` + + +```````````````````````````````` example +003. ok +. +
+
ok
+
+```````````````````````````````` + + +A start number may not be negative: + +```````````````````````````````` example +-1. not ok +. +
-1. not ok
+```````````````````````````````` + + + +2. **Item starting with indented code.** If a sequence of lines *Ls* + constitute a sequence of blocks *Bs* starting with an indented code + block, and *M* is a list marker of width *W* followed by + one space of indentation, then the result of prepending *M* and the + following space to the first line of *Ls*, and indenting subsequent lines + of *Ls* by *W + 1* spaces, is a list item with *Bs* as its contents. + If a line is empty, then it need not be indented. The type of the + list item (bullet or ordered) is determined by the type of its list + marker. If the list item is ordered, then it is also assigned a + start number, based on the ordered list marker. + +An indented code block will have to be preceded by four spaces of indentation +beyond the edge of the region where text will be included in the list item. +In the following case that is 6 spaces: + +```````````````````````````````` example +- foo + + bar +. +
+
+
foo
+
bar +
+
+
+```````````````````````````````` + + +And in this case it is 11 spaces: + +```````````````````````````````` example + 10. foo + + bar +. +
+
+
foo
+
bar +
+
+
+```````````````````````````````` + + +If the *first* block in the list item is an indented code block, +then by rule #2, the contents must be preceded by *one* space of indentation +after the list marker: + +```````````````````````````````` example + indented code + +paragraph + + more code +. +
indented code +
+
paragraph
+
more code +
+```````````````````````````````` + + +```````````````````````````````` example +1. indented code + + paragraph + + more code +. +
+
+
indented code +
+
paragraph
+
more code +
+
+
+```````````````````````````````` + + +Note that an additional space of indentation is interpreted as space +inside the code block: + +```````````````````````````````` example +1. indented code + + paragraph + + more code +. +
+
+
indented code +
+
paragraph
+
more code +
+
+
+```````````````````````````````` + + +Note that rules #1 and #2 only apply to two cases: (a) cases +in which the lines to be included in a list item begin with a +characer other than a space or tab, and (b) cases in which +they begin with an indented code +block. In a case like the following, where the first block begins with +three spaces of indentation, the rules do not allow us to form a list item by +indenting the whole thing and prepending a list marker: + +```````````````````````````````` example + foo + +bar +. +
foo
+
bar
+```````````````````````````````` + + +```````````````````````````````` example +- foo + + bar +. +
+
foo
+
+
bar
+```````````````````````````````` + + +This is not a significant restriction, because when a block is preceded by up to +three spaces of indentation, the indentation can always be removed without +a change in interpretation, allowing rule #1 to be applied. So, in +the above case: + +```````````````````````````````` example +- foo + + bar +. +
+
+
foo
+
bar
+
+
+```````````````````````````````` + + +3. **Item starting with a blank line.** If a sequence of lines *Ls* + starting with a single [blank line] constitute a (possibly empty) + sequence of blocks *Bs*, and *M* is a list marker of width *W*, + then the result of prepending *M* to the first line of *Ls*, and + preceding subsequent lines of *Ls* by *W + 1* spaces of indentation, is a + list item with *Bs* as its contents. + If a line is empty, then it need not be indented. The type of the + list item (bullet or ordered) is determined by the type of its list + marker. If the list item is ordered, then it is also assigned a + start number, based on the ordered list marker. + +Here are some list items that start with a blank line but are not empty: + +```````````````````````````````` example +- + foo +- + ``` + bar + ``` +- + baz +. +
+
foo
+
+
bar +
+
+
+
baz +
+
+
+```````````````````````````````` + +When the list item starts with a blank line, the number of spaces +following the list marker doesn't change the required indentation: + +```````````````````````````````` example +- + foo +. +
+
foo
+
+```````````````````````````````` + + +A list item can begin with at most one blank line. +In the following example, `foo` is not part of the list +item: + +```````````````````````````````` example +- + + foo +. +
+
+
+
foo
+```````````````````````````````` + + +Here is an empty bullet list item: + +```````````````````````````````` example +- foo +- +- bar +. +
+
foo
+
+
bar
+
+```````````````````````````````` + + +It does not matter whether there are spaces or tabs following the [list marker]: + +```````````````````````````````` example +- foo +- +- bar +. +
+
foo
+
+
bar
+
+```````````````````````````````` + + +Here is an empty ordered list item: + +```````````````````````````````` example +1. foo +2. +3. bar +. +
+
foo
+
+
bar
+
+```````````````````````````````` + + +A list may start or end with an empty list item: + +```````````````````````````````` example +* +. +
+
+
+```````````````````````````````` + +However, an empty list item cannot interrupt a paragraph: + +```````````````````````````````` example +foo +* + +foo +1. +. +
foo +*
+
foo +1.
+```````````````````````````````` + + +4. **Indentation.** If a sequence of lines *Ls* constitutes a list item + according to rule #1, #2, or #3, then the result of preceding each line + of *Ls* by up to three spaces of indentation (the same for each line) also + constitutes a list item with the same contents and attributes. If a line is + empty, then it need not be indented. + +Indented one space: + +```````````````````````````````` example + 1. A paragraph + with two lines. + + indented code + + > A block quote. +. +
+
+
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+
+
+```````````````````````````````` + + +Indented two spaces: + +```````````````````````````````` example + 1. A paragraph + with two lines. + + indented code + + > A block quote. +. +
+
+
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+
+
+```````````````````````````````` + + +Indented three spaces: + +```````````````````````````````` example + 1. A paragraph + with two lines. + + indented code + + > A block quote. +. +
+
+
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+
+
+```````````````````````````````` + + +Four spaces indent gives a code block: + +```````````````````````````````` example + 1. A paragraph + with two lines. + + indented code + + > A block quote. +. +
1. A paragraph + with two lines. + + indented code + + > A block quote. +
+```````````````````````````````` + + + +5. **Laziness.** If a string of lines *Ls* constitute a [list + item](#list-items) with contents *Bs*, then the result of deleting + some or all of the indentation from one or more lines in which the + next character other than a space or tab after the indentation is + [paragraph continuation text] is a + list item with the same contents and attributes. The unindented + lines are called + [lazy continuation line](@)s. + +Here is an example with [lazy continuation lines]: + +```````````````````````````````` example + 1. A paragraph +with two lines. + + indented code + + > A block quote. +. +
+
+
A paragraph +with two lines.
+
indented code +
+
+
A block quote.
+
+
+
+```````````````````````````````` + + +Indentation can be partially deleted: + +```````````````````````````````` example + 1. A paragraph + with two lines. +. +
+
A paragraph +with two lines.
+
+```````````````````````````````` + + +These examples show how laziness can work in nested structures: + +```````````````````````````````` example +> 1. > Blockquote +continued here. +. +
+
+
+
+
Blockquote +continued here.
+
+
+
+
+```````````````````````````````` + + +```````````````````````````````` example +> 1. > Blockquote +> continued here. +. +
+
+
+
+
Blockquote +continued here.
+
+
+
+
+```````````````````````````````` + + + +6. **That's all.** Nothing that is not counted as a list item by rules + #1--5 counts as a [list item](#list-items). + +The rules for sublists follow from the general rules +[above][List items]. A sublist must be indented the same number +of spaces of indentation a paragraph would need to be in order to be included +in the list item. + +So, in this case we need two spaces indent: + +```````````````````````````````` example +- foo + - bar + - baz + - boo +. +
+
foo +
+
bar +
+
baz +
+
boo
+
+
+
+
+
+
+
+```````````````````````````````` + + +One is not enough: + +```````````````````````````````` example +- foo + - bar + - baz + - boo +. +
+
foo
+
bar
+
baz
+
boo
+
+```````````````````````````````` + + +Here we need four, because the list marker is wider: + +```````````````````````````````` example +10) foo + - bar +. +
+
foo +
+
bar
+
+
+
+```````````````````````````````` + + +Three is not enough: + +```````````````````````````````` example +10) foo + - bar +. +
+
foo
+
+
+
bar
+
+```````````````````````````````` + + +A list may be the first block in a list item: + +```````````````````````````````` example +- - foo +. +
+
+
+
foo
+
+
+
+```````````````````````````````` + + +```````````````````````````````` example +1. - 2. foo +. +
+
+
+
+
+
foo
+
+
+
+
+
+```````````````````````````````` + + +A list item can contain a heading: + +```````````````````````````````` example +- # Foo +- Bar + --- + baz +. +
+
+
Foo
+
+
+
Bar
+baz
+
+```````````````````````````````` + + +### Motivation + +John Gruber's Markdown spec says the following about list items: + +1. "List markers typically start at the left margin, but may be indented + by up to three spaces. List markers must be followed by one or more + spaces or a tab." + +2. "To make lists look nice, you can wrap items with hanging indents.... + But if you don't want to, you don't have to." + +3. "List items may consist of multiple paragraphs. Each subsequent + paragraph in a list item must be indented by either 4 spaces or one + tab." + +4. "It looks nice if you indent every line of the subsequent paragraphs, + but here again, Markdown will allow you to be lazy." + +5. "To put a blockquote within a list item, the blockquote's `>` + delimiters need to be indented." + +6. "To put a code block within a list item, the code block needs to be + indented twice — 8 spaces or two tabs." + +These rules specify that a paragraph under a list item must be indented +four spaces (presumably, from the left margin, rather than the start of +the list marker, but this is not said), and that code under a list item +must be indented eight spaces instead of the usual four. They also say +that a block quote must be indented, but not by how much; however, the +example given has four spaces indentation. Although nothing is said +about other kinds of block-level content, it is certainly reasonable to +infer that *all* block elements under a list item, including other +lists, must be indented four spaces. This principle has been called the +*four-space rule*. + +The four-space rule is clear and principled, and if the reference +implementation `Markdown.pl` had followed it, it probably would have +become the standard. However, `Markdown.pl` allowed paragraphs and +sublists to start with only two spaces indentation, at least on the +outer level. Worse, its behavior was inconsistent: a sublist of an +outer-level list needed two spaces indentation, but a sublist of this +sublist needed three spaces. It is not surprising, then, that different +implementations of Markdown have developed very different rules for +determining what comes under a list item. (Pandoc and python-Markdown, +for example, stuck with Gruber's syntax description and the four-space +rule, while discount, redcarpet, marked, PHP Markdown, and others +followed `Markdown.pl`'s behavior more closely.) + +Unfortunately, given the divergences between implementations, there +is no way to give a spec for list items that will be guaranteed not +to break any existing documents. However, the spec given here should +correctly handle lists formatted with either the four-space rule or +the more forgiving `Markdown.pl` behavior, provided they are laid out +in a way that is natural for a human to read. + +The strategy here is to let the width and indentation of the list marker +determine the indentation necessary for blocks to fall under the list +item, rather than having a fixed and arbitrary number. The writer can +think of the body of the list item as a unit which gets indented to the +right enough to fit the list marker (and any indentation on the list +marker). (The laziness rule, #5, then allows continuation lines to be +unindented if needed.) + +This rule is superior, we claim, to any rule requiring a fixed level of +indentation from the margin. The four-space rule is clear but +unnatural. It is quite unintuitive that + +``` markdown +- foo + + bar + + - baz +``` + +should be parsed as two lists with an intervening paragraph, + +``` html +
+
foo
+
+
bar
+
+
baz
+
+``` + +as the four-space rule demands, rather than a single list, + +``` html +
+
+
foo
+
bar
+
+
baz
+
+
+
+``` + +The choice of four spaces is arbitrary. It can be learned, but it is +not likely to be guessed, and it trips up beginners regularly. + +Would it help to adopt a two-space rule? The problem is that such +a rule, together with the rule allowing up to three spaces of indentation for +the initial list marker, allows text that is indented *less than* the +original list marker to be included in the list item. For example, +`Markdown.pl` parses + +``` markdown + - one + + two +``` + +as a single list item, with `two` a continuation paragraph: + +``` html +
+
+
one
+
two
+
+
+``` + +and similarly + +``` markdown +> - one +> +> two +``` + +as + +``` html +
+
+
+
one
+
two
+
+
+
+``` + +This is extremely unintuitive. + +Rather than requiring a fixed indent from the margin, we could require +a fixed indent (say, two spaces, or even one space) from the list marker (which +may itself be indented). This proposal would remove the last anomaly +discussed. Unlike the spec presented above, it would count the following +as a list item with a subparagraph, even though the paragraph `bar` +is not indented as far as the first paragraph `foo`: + +``` markdown + 10. foo + + bar +``` + +Arguably this text does read like a list item with `bar` as a subparagraph, +which may count in favor of the proposal. However, on this proposal indented +code would have to be indented six spaces after the list marker. And this +would break a lot of existing Markdown, which has the pattern: + +``` markdown +1. foo + + indented code +``` + +where the code is indented eight spaces. The spec above, by contrast, will +parse this text as expected, since the code block's indentation is measured +from the beginning of `foo`. + +The one case that needs special treatment is a list item that *starts* +with indented code. How much indentation is required in that case, since +we don't have a "first paragraph" to measure from? Rule #2 simply stipulates +that in such cases, we require one space indentation from the list marker +(and then the normal four spaces for the indented code). This will match the +four-space rule in cases where the list marker plus its initial indentation +takes four spaces (a common case), but diverge in other cases. + +## Lists + +A [list](@) is a sequence of one or more +list items [of the same type]. The list items +may be separated by any number of blank lines. + +Two list items are [of the same type](@) +if they begin with a [list marker] of the same type. +Two list markers are of the +same type if (a) they are bullet list markers using the same character +(`-`, `+`, or `*`) or (b) they are ordered list numbers with the same +delimiter (either `.` or `)`). + +A list is an [ordered list](@) +if its constituent list items begin with +[ordered list markers], and a +[bullet list](@) if its constituent list +items begin with [bullet list markers]. + +The [start number](@) +of an [ordered list] is determined by the list number of +its initial list item. The numbers of subsequent list items are +disregarded. + +A list is [loose](@) if any of its constituent +list items are separated by blank lines, or if any of its constituent +list items directly contain two block-level elements with a blank line +between them. Otherwise a list is [tight](@). +(The difference in HTML output is that paragraphs in a loose list are +wrapped in `
` tags, while paragraphs in a tight list are not.) + +Changing the bullet or ordered list delimiter starts a new list: + +```````````````````````````````` example +- foo +- bar ++ baz +. +
+
foo
+
bar
+
+
+
baz
+
+```````````````````````````````` + + +```````````````````````````````` example +1. foo +2. bar +3) baz +. +
+
foo
+
bar
+
+
+
baz
+
+```````````````````````````````` + + +In CommonMark, a list can interrupt a paragraph. That is, +no blank line is needed to separate a paragraph from a following +list: + +```````````````````````````````` example +Foo +- bar +- baz +. +
Foo
+
+
bar
+
baz
+
+```````````````````````````````` + +`Markdown.pl` does not allow this, through fear of triggering a list +via a numeral in a hard-wrapped line: + +``` markdown +The number of windows in my house is +14. The number of doors is 6. +``` + +Oddly, though, `Markdown.pl` *does* allow a blockquote to +interrupt a paragraph, even though the same considerations might +apply. + +In CommonMark, we do allow lists to interrupt paragraphs, for +two reasons. First, it is natural and not uncommon for people +to start lists without blank lines: + +``` markdown +I need to buy +- new shoes +- a coat +- a plane ticket +``` + +Second, we are attracted to a + +> [principle of uniformity](@): +> if a chunk of text has a certain +> meaning, it will continue to have the same meaning when put into a +> container block (such as a list item or blockquote). + +(Indeed, the spec for [list items] and [block quotes] presupposes +this principle.) This principle implies that if + +``` markdown + * I need to buy + - new shoes + - a coat + - a plane ticket +``` + +is a list item containing a paragraph followed by a nested sublist, +as all Markdown implementations agree it is (though the paragraph +may be rendered without `
` tags, since the list is "tight"), +then + +``` markdown +I need to buy +- new shoes +- a coat +- a plane ticket +``` + +by itself should be a paragraph followed by a nested sublist. + +Since it is well established Markdown practice to allow lists to +interrupt paragraphs inside list items, the [principle of +uniformity] requires us to allow this outside list items as +well. ([reStructuredText](http://docutils.sourceforge.net/rst.html) +takes a different approach, requiring blank lines before lists +even inside other list items.) + +In order to solve of unwanted lists in paragraphs with +hard-wrapped numerals, we allow only lists starting with `1` to +interrupt paragraphs. Thus, + +```````````````````````````````` example +The number of windows in my house is +14. The number of doors is 6. +. +
The number of windows in my house is +14. The number of doors is 6.
+```````````````````````````````` + +We may still get an unintended result in cases like + +```````````````````````````````` example +The number of windows in my house is +1. The number of doors is 6. +. +
The number of windows in my house is
+
+
The number of doors is 6.
+
+```````````````````````````````` + +but this rule should prevent most spurious list captures. + +There can be any number of blank lines between items: + +```````````````````````````````` example +- foo + +- bar + + +- baz +. +
+
+
foo
+
+
+
bar
+
+
+
baz
+
+
+```````````````````````````````` + +```````````````````````````````` example +- foo + - bar + - baz + + + bim +. +
+
foo +
+
bar +
+
+
baz
+
bim
+
+
+
+
+
+
+```````````````````````````````` + + +To separate consecutive lists of the same type, or to separate a +list from an indented code block that would otherwise be parsed +as a subparagraph of the final list item, you can insert a blank HTML +comment: + +```````````````````````````````` example +- foo +- bar + + + +- baz +- bim +. +
+
foo
+
bar
+
+ +
+
baz
+
bim
+
+```````````````````````````````` + + +```````````````````````````````` example +- foo + + notcode + +- foo + + + + code +. +
+
+
foo
+
notcode
+
+
+
foo
+
+
+ +
code +
+```````````````````````````````` + + +List items need not be indented to the same level. The following +list items will be treated as items at the same list level, +since none is indented enough to belong to the previous list +item: + +```````````````````````````````` example +- a + - b + - c + - d + - e + - f +- g +. +
+
a
+
b
+
c
+
d
+
e
+
f
+
g
+
+```````````````````````````````` + + +```````````````````````````````` example +1. a + + 2. b + + 3. c +. +
+
+
a
+
+
+
b
+
+
+
c
+
+
+```````````````````````````````` + +Note, however, that list items may not be preceded by more than +three spaces of indentation. Here `- e` is treated as a paragraph continuation +line, because it is indented more than three spaces: + +```````````````````````````````` example +- a + - b + - c + - d + - e +. +
+
a
+
b
+
c
+
d +- e
+
+```````````````````````````````` + +And here, `3. c` is treated as in indented code block, +because it is indented four spaces and preceded by a +blank line. + +```````````````````````````````` example +1. a + + 2. b + + 3. c +. +
+
+
a
+
+
+
b
+
+
+
3. c +
+```````````````````````````````` + + +This is a loose list, because there is a blank line between +two of the list items: + +```````````````````````````````` example +- a +- b + +- c +. +
+
+
a
+
+
+
b
+
+
+
c
+
+
+```````````````````````````````` + + +So is this, with a empty second item: + +```````````````````````````````` example +* a +* + +* c +. +
+
+
a
+
+
+
+
c
+
+
+```````````````````````````````` + + +These are loose lists, even though there are no blank lines between the items, +because one of the items directly contains two block-level elements +with a blank line between them: + +```````````````````````````````` example +- a +- b + + c +- d +. +
+
+
a
+
+
+
b
+
c
+
+
+
d
+
+
+```````````````````````````````` + + +```````````````````````````````` example +- a +- b + + [ref]: /url +- d +. +
+
+
a
+
+
+
b
+
+
+
d
+
+
+```````````````````````````````` + + +This is a tight list, because the blank lines are in a code block: + +```````````````````````````````` example +- a +- ``` + b + + + ``` +- c +. +
+
a
+
+
b + + +
+
+
c
+
+```````````````````````````````` + + +This is a tight list, because the blank line is between two +paragraphs of a sublist. So the sublist is loose while +the outer list is tight: + +```````````````````````````````` example +- a + - b + + c +- d +. +
+
a +
+
+
b
+
c
+
+
+
+
d
+
+```````````````````````````````` + + +This is a tight list, because the blank line is inside the +block quote: + +```````````````````````````````` example +* a + > b + > +* c +. +
+
a +
+
b
+
+
+
c
+
+```````````````````````````````` + + +This list is tight, because the consecutive block elements +are not separated by blank lines: + +```````````````````````````````` example +- a + > b + ``` + c + ``` +- d +. +
+
a +
+
b
+
+
c +
+
+
d
+
+```````````````````````````````` + + +A single-paragraph list is tight: + +```````````````````````````````` example +- a +. +
+
a
+
+```````````````````````````````` + + +```````````````````````````````` example +- a + - b +. +
+
a +
+
b
+
+
+
+```````````````````````````````` + + +This list is loose, because of the blank line between the +two block elements in the list item: + +```````````````````````````````` example +1. ``` + foo + ``` + + bar +. +
+
+
foo +
+
bar
+
+
+```````````````````````````````` + + +Here the outer list is loose, the inner list tight: + +```````````````````````````````` example +* foo + * bar + + baz +. +
+
+
foo
+
+
bar
+
+
baz
+
+
+```````````````````````````````` + + +```````````````````````````````` example +- a + - b + - c + +- d + - e + - f +. +
+
+
a
+
+
b
+
c
+
+
+
+
d
+
+
e
+
f
+
+
+
+```````````````````````````````` + + +# Inlines + +Inlines are parsed sequentially from the beginning of the character +stream to the end (left to right, in left-to-right languages). +Thus, for example, in + +```````````````````````````````` example +`hi`lo` +. +
hilo`
+```````````````````````````````` + +`hi` is parsed as code, leaving the backtick at the end as a literal +backtick. + + + +## Code spans + +A [backtick string](@) +is a string of one or more backtick characters (`` ` ``) that is neither +preceded nor followed by a backtick. + +A [code span](@) begins with a backtick string and ends with +a backtick string of equal length. The contents of the code span are +the characters between these two backtick strings, normalized in the +following ways: + +- First, [line endings] are converted to [spaces]. +- If the resulting string both begins *and* ends with a [space] + character, but does not consist entirely of [space] + characters, a single [space] character is removed from the + front and back. This allows you to include code that begins + or ends with backtick characters, which must be separated by + whitespace from the opening or closing backtick strings. + +This is a simple code span: + +```````````````````````````````` example +`foo` +. +
foo
+```````````````````````````````` + + +Here two backticks are used, because the code contains a backtick. +This example also illustrates stripping of a single leading and +trailing space: + +```````````````````````````````` example +`` foo ` bar `` +. +
foo ` bar
+```````````````````````````````` + + +This example shows the motivation for stripping leading and trailing +spaces: + +```````````````````````````````` example +` `` ` +. +
``
+```````````````````````````````` + +Note that only *one* space is stripped: + +```````````````````````````````` example +` `` ` +. +
``
+```````````````````````````````` + +The stripping only happens if the space is on both +sides of the string: + +```````````````````````````````` example +` a` +. +
a
+```````````````````````````````` + +Only [spaces], and not [unicode whitespace] in general, are +stripped in this way: + +```````````````````````````````` example +` b ` +. +
b
+```````````````````````````````` + +No stripping occurs if the code span contains only spaces: + +```````````````````````````````` example +` ` +` ` +. +
+
+```````````````````````````````` + + +[Line endings] are treated like spaces: + +```````````````````````````````` example +`` +foo +bar +baz +`` +. +
foo bar baz
+```````````````````````````````` + +```````````````````````````````` example +`` +foo +`` +. +
foo
+```````````````````````````````` + + +Interior spaces are not collapsed: + +```````````````````````````````` example +`foo bar +baz` +. +
foo bar baz
+```````````````````````````````` + +Note that browsers will typically collapse consecutive spaces +when rendering `` elements, so it is recommended that +the following CSS be used: + + code{white-space: pre-wrap;} + + +Note that backslash escapes do not work in code spans. All backslashes +are treated literally: + +```````````````````````````````` example +`foo\`bar` +. +foo\bar` +```````````````````````````````` + + +Backslash escapes are never needed, because one can always choose a +string of *n* backtick characters as delimiters, where the code does +not contain any strings of exactly *n* backtick characters. + +```````````````````````````````` example +``foo`bar`` +. +foo`bar +```````````````````````````````` + +```````````````````````````````` example +` foo `` bar ` +. +foo `` bar +```````````````````````````````` + + +Code span backticks have higher precedence than any other inline +constructs except HTML tags and autolinks. Thus, for example, this is +not parsed as emphasized text, since the second `*` is part of a code +span: + +```````````````````````````````` example +*foo`*` +. +*foo* +```````````````````````````````` + + +And this is not parsed as a link: + +```````````````````````````````` example +[not a `link](/foo`) +. +[not a link](/foo) +```````````````````````````````` + + +Code spans, HTML tags, and autolinks have the same precedence. +Thus, this is code: + +```````````````````````````````` example +`` +. +<a href="">` +```````````````````````````````` + + +But this is an HTML tag: + +```````````````````````````````` example +` +. +` +```````````````````````````````` + + +And this is code: + +```````````````````````````````` example +`` +. +<http://foo.bar.baz>` +```````````````````````````````` + + +But this is an autolink: + +```````````````````````````````` example +` +. +http://foo.bar.`baz` +```````````````````````````````` + + +When a backtick string is not closed by a matching backtick string, +we just have literal backticks: + +```````````````````````````````` example +```foo`` +. +```foo`` +```````````````````````````````` + + +```````````````````````````````` example +`foo +. +`foo +```````````````````````````````` + +The following case also illustrates the need for opening and +closing backtick strings to be equal in length: + +```````````````````````````````` example +`foo``bar`` +. +`foobar +```````````````````````````````` + + +## Emphasis and strong emphasis + +John Gruber's original [Markdown syntax +description](http://daringfireball.net/projects/markdown/syntax#em) says: + +> Markdown treats asterisks (`*`) and underscores (`_`) as indicators of +> emphasis. Text wrapped with one `*` or `_` will be wrapped with an HTML +> `` tag; double `*`'s or `_`'s will be wrapped with an HTML `` +> tag. + +This is enough for most users, but these rules leave much undecided, +especially when it comes to nested emphasis. The original +`Markdown.pl` test suite makes it clear that triple `***` and +`___` delimiters can be used for strong emphasis, and most +implementations have also allowed the following patterns: + +``` markdown +***strong emph*** +***strong** in emph* +***emph* in strong** +**in strong *emph*** +*in emph **strong*** +``` + +The following patterns are less widely supported, but the intent +is clear and they are useful (especially in contexts like bibliography +entries): + +``` markdown +*emph *with emph* in it* +**strong **with strong** in it** +``` + +Many implementations have also restricted intraword emphasis to +the `*` forms, to avoid unwanted emphasis in words containing +internal underscores. (It is best practice to put these in code +spans, but users often do not.) + +``` markdown +internal emphasis: foo*bar*baz +no emphasis: foo_bar_baz +``` + +The rules given below capture all of these patterns, while allowing +for efficient parsing strategies that do not backtrack. + +First, some definitions. A [delimiter run](@) is either +a sequence of one or more `*` characters that is not preceded or +followed by a non-backslash-escaped `*` character, or a sequence +of one or more `_` characters that is not preceded or followed by +a non-backslash-escaped `_` character. + +A [left-flanking delimiter run](@) is +a [delimiter run] that is (1) not followed by [Unicode whitespace], +and either (2a) not followed by a [Unicode punctuation character], or +(2b) followed by a [Unicode punctuation character] and +preceded by [Unicode whitespace] or a [Unicode punctuation character]. +For purposes of this definition, the beginning and the end of +the line count as Unicode whitespace. + +A [right-flanking delimiter run](@) is +a [delimiter run] that is (1) not preceded by [Unicode whitespace], +and either (2a) not preceded by a [Unicode punctuation character], or +(2b) preceded by a [Unicode punctuation character] and +followed by [Unicode whitespace] or a [Unicode punctuation character]. +For purposes of this definition, the beginning and the end of +the line count as Unicode whitespace. + +Here are some examples of delimiter runs. + + - left-flanking but not right-flanking: + + ``` + ***abc + _abc + **"abc" + _"abc" + ``` + + - right-flanking but not left-flanking: + + ``` + abc*** + abc_ + "abc"** + "abc"_ + ``` + + - Both left and right-flanking: + + ``` + abc***def + "abc"_"def" + ``` + + - Neither left nor right-flanking: + + ``` + abc *** def + a _ b + ``` + +(The idea of distinguishing left-flanking and right-flanking +delimiter runs based on the character before and the character +after comes from Roopesh Chander's +[vfmd](http://www.vfmd.org/vfmd-spec/specification/#procedure-for-identifying-emphasis-tags). +vfmd uses the terminology "emphasis indicator string" instead of "delimiter +run," and its rules for distinguishing left- and right-flanking runs +are a bit more complex than the ones given here.) + +The following rules define emphasis and strong emphasis: + +1. A single `*` character [can open emphasis](@) + iff (if and only if) it is part of a [left-flanking delimiter run]. + +2. A single `_` character [can open emphasis] iff + it is part of a [left-flanking delimiter run] + and either (a) not part of a [right-flanking delimiter run] + or (b) part of a [right-flanking delimiter run] + preceded by a [Unicode punctuation character]. + +3. A single `*` character [can close emphasis](@) + iff it is part of a [right-flanking delimiter run]. + +4. A single `_` character [can close emphasis] iff + it is part of a [right-flanking delimiter run] + and either (a) not part of a [left-flanking delimiter run] + or (b) part of a [left-flanking delimiter run] + followed by a [Unicode punctuation character]. + +5. A double `**` [can open strong emphasis](@) + iff it is part of a [left-flanking delimiter run]. + +6. A double `__` [can open strong emphasis] iff + it is part of a [left-flanking delimiter run] + and either (a) not part of a [right-flanking delimiter run] + or (b) part of a [right-flanking delimiter run] + preceded by a [Unicode punctuation character]. + +7. A double `**` [can close strong emphasis](@) + iff it is part of a [right-flanking delimiter run]. + +8. A double `__` [can close strong emphasis] iff + it is part of a [right-flanking delimiter run] + and either (a) not part of a [left-flanking delimiter run] + or (b) part of a [left-flanking delimiter run] + followed by a [Unicode punctuation character]. + +9. Emphasis begins with a delimiter that [can open emphasis] and ends + with a delimiter that [can close emphasis], and that uses the same + character (`_` or `*`) as the opening delimiter. The + opening and closing delimiters must belong to separate + [delimiter runs]. If one of the delimiters can both + open and close emphasis, then the sum of the lengths of the + delimiter runs containing the opening and closing delimiters + must not be a multiple of 3 unless both lengths are + multiples of 3. + +10. Strong emphasis begins with a delimiter that + [can open strong emphasis] and ends with a delimiter that + [can close strong emphasis], and that uses the same character + (`_` or `*`) as the opening delimiter. The + opening and closing delimiters must belong to separate + [delimiter runs]. If one of the delimiters can both open + and close strong emphasis, then the sum of the lengths of + the delimiter runs containing the opening and closing + delimiters must not be a multiple of 3 unless both lengths + are multiples of 3. + +11. A literal `*` character cannot occur at the beginning or end of + `*`-delimited emphasis or `**`-delimited strong emphasis, unless it + is backslash-escaped. + +12. A literal `_` character cannot occur at the beginning or end of + `_`-delimited emphasis or `__`-delimited strong emphasis, unless it + is backslash-escaped. + +Where rules 1--12 above are compatible with multiple parsings, +the following principles resolve ambiguity: + +13. The number of nestings should be minimized. Thus, for example, + an interpretation `...` is always preferred to + `...`. + +14. An interpretation `...` is always + preferred to `...`. + +15. When two potential emphasis or strong emphasis spans overlap, + so that the second begins before the first ends and ends after + the first ends, the first takes precedence. Thus, for example, + `*foo _bar* baz_` is parsed as `foo _bar baz_` rather + than `*foo bar* baz`. + +16. When there are two potential emphasis or strong emphasis spans + with the same closing delimiter, the shorter one (the one that + opens later) takes precedence. Thus, for example, + `**foo **bar baz**` is parsed as `**foo bar baz` + rather than `foo **bar baz`. + +17. Inline code spans, links, images, and HTML tags group more tightly + than emphasis. So, when there is a choice between an interpretation + that contains one of these elements and one that does not, the + former always wins. Thus, for example, `*[foo*](bar)` is + parsed as `*foo*` rather than as + `[foo](bar)`. + +These rules can be illustrated through a series of examples. + +Rule 1: + +```````````````````````````````` example +*foo bar* +. +foo bar +```````````````````````````````` + + +This is not emphasis, because the opening `*` is followed by +whitespace, and hence not part of a [left-flanking delimiter run]: + +```````````````````````````````` example +a * foo bar* +. +a * foo bar* +```````````````````````````````` + + +This is not emphasis, because the opening `*` is preceded +by an alphanumeric and followed by punctuation, and hence +not part of a [left-flanking delimiter run]: + +```````````````````````````````` example +a*"foo"* +. +a*"foo"* +```````````````````````````````` + + +Unicode nonbreaking spaces count as whitespace, too: + +```````````````````````````````` example +* a * +. +* a * +```````````````````````````````` + + +Intraword emphasis with `*` is permitted: + +```````````````````````````````` example +foo*bar* +. +foobar +```````````````````````````````` + + +```````````````````````````````` example +5*6*78 +. +5678 +```````````````````````````````` + + +Rule 2: + +```````````````````````````````` example +_foo bar_ +. +foo bar +```````````````````````````````` + + +This is not emphasis, because the opening `_` is followed by +whitespace: + +```````````````````````````````` example +_ foo bar_ +. +_ foo bar_ +```````````````````````````````` + + +This is not emphasis, because the opening `_` is preceded +by an alphanumeric and followed by punctuation: + +```````````````````````````````` example +a_"foo"_ +. +a_"foo"_ +```````````````````````````````` + + +Emphasis with `_` is not allowed inside words: + +```````````````````````````````` example +foo_bar_ +. +foo_bar_ +```````````````````````````````` + + +```````````````````````````````` example +5_6_78 +. +5_6_78 +```````````````````````````````` + + +```````````````````````````````` example +пристаням_стремятся_ +. +пристаням_стремятся_ +```````````````````````````````` + + +Here `_` does not generate emphasis, because the first delimiter run +is right-flanking and the second left-flanking: + +```````````````````````````````` example +aa_"bb"_cc +. +aa_"bb"_cc +```````````````````````````````` + + +This is emphasis, even though the opening delimiter is +both left- and right-flanking, because it is preceded by +punctuation: + +```````````````````````````````` example +foo-_(bar)_ +. +foo-(bar) +```````````````````````````````` + + +Rule 3: + +This is not emphasis, because the closing delimiter does +not match the opening delimiter: + +```````````````````````````````` example +_foo* +. +_foo* +```````````````````````````````` + + +This is not emphasis, because the closing `*` is preceded by +whitespace: + +```````````````````````````````` example +*foo bar * +. +*foo bar * +```````````````````````````````` + + +A line ending also counts as whitespace: + +```````````````````````````````` example +*foo bar +* +. +*foo bar +* +```````````````````````````````` + + +This is not emphasis, because the second `*` is +preceded by punctuation and followed by an alphanumeric +(hence it is not part of a [right-flanking delimiter run]: + +```````````````````````````````` example +*(*foo) +. +*(*foo) +```````````````````````````````` + + +The point of this restriction is more easily appreciated +with this example: + +```````````````````````````````` example +*(*foo*)* +. +(foo) +```````````````````````````````` + + +Intraword emphasis with `*` is allowed: + +```````````````````````````````` example +*foo*bar +. +foobar +```````````````````````````````` + + + +Rule 4: + +This is not emphasis, because the closing `_` is preceded by +whitespace: + +```````````````````````````````` example +_foo bar _ +. +_foo bar _ +```````````````````````````````` + + +This is not emphasis, because the second `_` is +preceded by punctuation and followed by an alphanumeric: + +```````````````````````````````` example +_(_foo) +. +_(_foo) +```````````````````````````````` + + +This is emphasis within emphasis: + +```````````````````````````````` example +_(_foo_)_ +. +(foo) +```````````````````````````````` + + +Intraword emphasis is disallowed for `_`: + +```````````````````````````````` example +_foo_bar +. +_foo_bar +```````````````````````````````` + + +```````````````````````````````` example +_пристаням_стремятся +. +_пристаням_стремятся +```````````````````````````````` + + +```````````````````````````````` example +_foo_bar_baz_ +. +foo_bar_baz +```````````````````````````````` + + +This is emphasis, even though the closing delimiter is +both left- and right-flanking, because it is followed by +punctuation: + +```````````````````````````````` example +_(bar)_. +. +(bar). +```````````````````````````````` + + +Rule 5: + +```````````````````````````````` example +**foo bar** +. +foo bar +```````````````````````````````` + + +This is not strong emphasis, because the opening delimiter is +followed by whitespace: + +```````````````````````````````` example +** foo bar** +. +** foo bar** +```````````````````````````````` + + +This is not strong emphasis, because the opening `**` is preceded +by an alphanumeric and followed by punctuation, and hence +not part of a [left-flanking delimiter run]: + +```````````````````````````````` example +a**"foo"** +. +a**"foo"** +```````````````````````````````` + + +Intraword strong emphasis with `**` is permitted: + +```````````````````````````````` example +foo**bar** +. +foobar +```````````````````````````````` + + +Rule 6: + +```````````````````````````````` example +__foo bar__ +. +foo bar +```````````````````````````````` + + +This is not strong emphasis, because the opening delimiter is +followed by whitespace: + +```````````````````````````````` example +__ foo bar__ +. +__ foo bar__ +```````````````````````````````` + + +A line ending counts as whitespace: +```````````````````````````````` example +__ +foo bar__ +. +__ +foo bar__ +```````````````````````````````` + + +This is not strong emphasis, because the opening `__` is preceded +by an alphanumeric and followed by punctuation: + +```````````````````````````````` example +a__"foo"__ +. +a__"foo"__ +```````````````````````````````` + + +Intraword strong emphasis is forbidden with `__`: + +```````````````````````````````` example +foo__bar__ +. +foo__bar__ +```````````````````````````````` + + +```````````````````````````````` example +5__6__78 +. +5__6__78 +```````````````````````````````` + + +```````````````````````````````` example +пристаням__стремятся__ +. +пристаням__стремятся__ +```````````````````````````````` + + +```````````````````````````````` example +__foo, __bar__, baz__ +. +foo, bar, baz +```````````````````````````````` + + +This is strong emphasis, even though the opening delimiter is +both left- and right-flanking, because it is preceded by +punctuation: + +```````````````````````````````` example +foo-__(bar)__ +. +foo-(bar) +```````````````````````````````` + + + +Rule 7: + +This is not strong emphasis, because the closing delimiter is preceded +by whitespace: + +```````````````````````````````` example +**foo bar ** +. +**foo bar ** +```````````````````````````````` + + +(Nor can it be interpreted as an emphasized `*foo bar *`, because of +Rule 11.) + +This is not strong emphasis, because the second `**` is +preceded by punctuation and followed by an alphanumeric: + +```````````````````````````````` example +**(**foo) +. +**(**foo) +```````````````````````````````` + + +The point of this restriction is more easily appreciated +with these examples: + +```````````````````````````````` example +*(**foo**)* +. +(foo) +```````````````````````````````` + + +```````````````````````````````` example +**Gomphocarpus (*Gomphocarpus physocarpus*, syn. +*Asclepias physocarpa*)** +. +Gomphocarpus (Gomphocarpus physocarpus, syn. +Asclepias physocarpa) +```````````````````````````````` + + +```````````````````````````````` example +**foo "*bar*" foo** +. +foo "bar" foo +```````````````````````````````` + + +Intraword emphasis: + +```````````````````````````````` example +**foo**bar +. +foobar +```````````````````````````````` + + +Rule 8: + +This is not strong emphasis, because the closing delimiter is +preceded by whitespace: + +```````````````````````````````` example +__foo bar __ +. +__foo bar __ +```````````````````````````````` + + +This is not strong emphasis, because the second `__` is +preceded by punctuation and followed by an alphanumeric: + +```````````````````````````````` example +__(__foo) +. +__(__foo) +```````````````````````````````` + + +The point of this restriction is more easily appreciated +with this example: + +```````````````````````````````` example +_(__foo__)_ +. +(foo) +```````````````````````````````` + + +Intraword strong emphasis is forbidden with `__`: + +```````````````````````````````` example +__foo__bar +. +__foo__bar +```````````````````````````````` + + +```````````````````````````````` example +__пристаням__стремятся +. +__пристаням__стремятся +```````````````````````````````` + + +```````````````````````````````` example +__foo__bar__baz__ +. +foo__bar__baz +```````````````````````````````` + + +This is strong emphasis, even though the closing delimiter is +both left- and right-flanking, because it is followed by +punctuation: + +```````````````````````````````` example +__(bar)__. +. +(bar). +```````````````````````````````` + + +Rule 9: + +Any nonempty sequence of inline elements can be the contents of an +emphasized span. + +```````````````````````````````` example +*foo [bar](/url)* +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +*foo +bar* +. +foo +bar +```````````````````````````````` + + +In particular, emphasis and strong emphasis can be nested +inside emphasis: + +```````````````````````````````` example +_foo __bar__ baz_ +. +foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +_foo _bar_ baz_ +. +foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +__foo_ bar_ +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +*foo *bar** +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +*foo **bar** baz* +. +foo bar baz +```````````````````````````````` + +```````````````````````````````` example +*foo**bar**baz* +. +foobarbaz +```````````````````````````````` + +Note that in the preceding case, the interpretation + +``` markdown +foobarbaz +``` + + +is precluded by the condition that a delimiter that +can both open and close (like the `*` after `foo`) +cannot form emphasis if the sum of the lengths of +the delimiter runs containing the opening and +closing delimiters is a multiple of 3 unless +both lengths are multiples of 3. + + +For the same reason, we don't get two consecutive +emphasis sections in this example: + +```````````````````````````````` example +*foo**bar* +. +foo**bar +```````````````````````````````` + + +The same condition ensures that the following +cases are all strong emphasis nested inside +emphasis, even when the interior whitespace is +omitted: + + +```````````````````````````````` example +***foo** bar* +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +*foo **bar*** +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +*foo**bar*** +. +foobar +```````````````````````````````` + + +When the lengths of the interior closing and opening +delimiter runs are *both* multiples of 3, though, +they can match to create emphasis: + +```````````````````````````````` example +foo***bar***baz +. +foobarbaz +```````````````````````````````` + +```````````````````````````````` example +foo******bar*********baz +. +foobar***baz +```````````````````````````````` + + +Indefinite levels of nesting are possible: + +```````````````````````````````` example +*foo **bar *baz* bim** bop* +. +foo bar baz bim bop +```````````````````````````````` + + +```````````````````````````````` example +*foo [*bar*](/url)* +. +foo bar +```````````````````````````````` + + +There can be no empty emphasis or strong emphasis: + +```````````````````````````````` example +** is not an empty emphasis +. +** is not an empty emphasis +```````````````````````````````` + + +```````````````````````````````` example +**** is not an empty strong emphasis +. +**** is not an empty strong emphasis +```````````````````````````````` + + + +Rule 10: + +Any nonempty sequence of inline elements can be the contents of an +strongly emphasized span. + +```````````````````````````````` example +**foo [bar](/url)** +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +**foo +bar** +. +foo +bar +```````````````````````````````` + + +In particular, emphasis and strong emphasis can be nested +inside strong emphasis: + +```````````````````````````````` example +__foo _bar_ baz__ +. +foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +__foo __bar__ baz__ +. +foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +____foo__ bar__ +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +**foo **bar**** +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +**foo *bar* baz** +. +foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +**foo*bar*baz** +. +foobarbaz +```````````````````````````````` + + +```````````````````````````````` example +***foo* bar** +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +**foo *bar*** +. +foo bar +```````````````````````````````` + + +Indefinite levels of nesting are possible: + +```````````````````````````````` example +**foo *bar **baz** +bim* bop** +. +foo bar baz +bim bop +```````````````````````````````` + + +```````````````````````````````` example +**foo [*bar*](/url)** +. +foo bar +```````````````````````````````` + + +There can be no empty emphasis or strong emphasis: + +```````````````````````````````` example +__ is not an empty emphasis +. +__ is not an empty emphasis +```````````````````````````````` + + +```````````````````````````````` example +____ is not an empty strong emphasis +. +____ is not an empty strong emphasis +```````````````````````````````` + + + +Rule 11: + +```````````````````````````````` example +foo *** +. +foo *** +```````````````````````````````` + + +```````````````````````````````` example +foo *\** +. +foo * +```````````````````````````````` + + +```````````````````````````````` example +foo *_* +. +foo _ +```````````````````````````````` + + +```````````````````````````````` example +foo ***** +. +foo ***** +```````````````````````````````` + + +```````````````````````````````` example +foo **\*** +. +foo * +```````````````````````````````` + + +```````````````````````````````` example +foo **_** +. +foo _ +```````````````````````````````` + + +Note that when delimiters do not match evenly, Rule 11 determines +that the excess literal `*` characters will appear outside of the +emphasis, rather than inside it: + +```````````````````````````````` example +**foo* +. +*foo +```````````````````````````````` + + +```````````````````````````````` example +*foo** +. +foo* +```````````````````````````````` + + +```````````````````````````````` example +***foo** +. +*foo +```````````````````````````````` + + +```````````````````````````````` example +****foo* +. +***foo +```````````````````````````````` + + +```````````````````````````````` example +**foo*** +. +foo* +```````````````````````````````` + + +```````````````````````````````` example +*foo**** +. +foo*** +```````````````````````````````` + + + +Rule 12: + +```````````````````````````````` example +foo ___ +. +foo ___ +```````````````````````````````` + + +```````````````````````````````` example +foo _\__ +. +foo _ +```````````````````````````````` + + +```````````````````````````````` example +foo _*_ +. +foo * +```````````````````````````````` + + +```````````````````````````````` example +foo _____ +. +foo _____ +```````````````````````````````` + + +```````````````````````````````` example +foo __\___ +. +foo _ +```````````````````````````````` + + +```````````````````````````````` example +foo __*__ +. +foo * +```````````````````````````````` + + +```````````````````````````````` example +__foo_ +. +_foo +```````````````````````````````` + + +Note that when delimiters do not match evenly, Rule 12 determines +that the excess literal `_` characters will appear outside of the +emphasis, rather than inside it: + +```````````````````````````````` example +_foo__ +. +foo_ +```````````````````````````````` + + +```````````````````````````````` example +___foo__ +. +_foo +```````````````````````````````` + + +```````````````````````````````` example +____foo_ +. +___foo +```````````````````````````````` + + +```````````````````````````````` example +__foo___ +. +foo_ +```````````````````````````````` + + +```````````````````````````````` example +_foo____ +. +foo___ +```````````````````````````````` + + +Rule 13 implies that if you want emphasis nested directly inside +emphasis, you must use different delimiters: + +```````````````````````````````` example +**foo** +. +foo +```````````````````````````````` + + +```````````````````````````````` example +*_foo_* +. +foo +```````````````````````````````` + + +```````````````````````````````` example +__foo__ +. +foo +```````````````````````````````` + + +```````````````````````````````` example +_*foo*_ +. +foo +```````````````````````````````` + + +However, strong emphasis within strong emphasis is possible without +switching delimiters: + +```````````````````````````````` example +****foo**** +. +foo +```````````````````````````````` + + +```````````````````````````````` example +____foo____ +. +foo +```````````````````````````````` + + + +Rule 13 can be applied to arbitrarily long sequences of +delimiters: + +```````````````````````````````` example +******foo****** +. +foo +```````````````````````````````` + + +Rule 14: + +```````````````````````````````` example +***foo*** +. +foo +```````````````````````````````` + + +```````````````````````````````` example +_____foo_____ +. +foo +```````````````````````````````` + + +Rule 15: + +```````````````````````````````` example +*foo _bar* baz_ +. +foo _bar baz_ +```````````````````````````````` + + +```````````````````````````````` example +*foo __bar *baz bim__ bam* +. +foo bar *baz bim bam +```````````````````````````````` + + +Rule 16: + +```````````````````````````````` example +**foo **bar baz** +. +**foo bar baz +```````````````````````````````` + + +```````````````````````````````` example +*foo *bar baz* +. +*foo bar baz +```````````````````````````````` + + +Rule 17: + +```````````````````````````````` example +*[bar*](/url) +. +*bar* +```````````````````````````````` + + +```````````````````````````````` example +_foo [bar_](/url) +. +_foo bar_ +```````````````````````````````` + + +```````````````````````````````` example +* +. +* +```````````````````````````````` + + +```````````````````````````````` example +** +. +** +```````````````````````````````` + + +```````````````````````````````` example +__ +. +__ +```````````````````````````````` + + +```````````````````````````````` example +*a `*`* +. +a * +```````````````````````````````` + + +```````````````````````````````` example +_a `_`_ +. +a _ +```````````````````````````````` + + +```````````````````````````````` example +**a +. +**ahttp://foo.bar/?q=** +```````````````````````````````` + + +```````````````````````````````` example +__a +. +__ahttp://foo.bar/?q=__ +```````````````````````````````` + + + +## Links + +A link contains [link text] (the visible text), a [link destination] +(the URI that is the link destination), and optionally a [link title]. +There are two basic kinds of links in Markdown. In [inline links] the +destination and title are given immediately after the link text. In +[reference links] the destination and title are defined elsewhere in +the document. + +A [link text](@) consists of a sequence of zero or more +inline elements enclosed by square brackets (`[` and `]`). The +following rules apply: + +- Links may not contain other links, at any level of nesting. If + multiple otherwise valid link definitions appear nested inside each + other, the inner-most definition is used. + +- Brackets are allowed in the [link text] only if (a) they + are backslash-escaped or (b) they appear as a matched pair of brackets, + with an open bracket `[`, a sequence of zero or more inlines, and + a close bracket `]`. + +- Backtick [code spans], [autolinks], and raw [HTML tags] bind more tightly + than the brackets in link text. Thus, for example, + `` [foo`]` `` could not be a link text, since the second `]` + is part of a code span. + +- The brackets in link text bind more tightly than markers for + [emphasis and strong emphasis]. Thus, for example, `*[foo*](url)` is a link. + +A [link destination](@) consists of either + +- a sequence of zero or more characters between an opening `<` and a + closing `>` that contains no line endings or unescaped + `<` or `>` characters, or + +- a nonempty sequence of characters that does not start with `<`, + does not include [ASCII control characters][ASCII control character] + or [space] character, and includes parentheses only if (a) they are + backslash-escaped or (b) they are part of a balanced pair of + unescaped parentheses. + (Implementations may impose limits on parentheses nesting to + avoid performance issues, but at least three levels of nesting + should be supported.) + +A [link title](@) consists of either + +- a sequence of zero or more characters between straight double-quote + characters (`"`), including a `"` character only if it is + backslash-escaped, or + +- a sequence of zero or more characters between straight single-quote + characters (`'`), including a `'` character only if it is + backslash-escaped, or + +- a sequence of zero or more characters between matching parentheses + (`(...)`), including a `(` or `)` character only if it is + backslash-escaped. + +Although [link titles] may span multiple lines, they may not contain +a [blank line]. + +An [inline link](@) consists of a [link text] followed immediately +by a left parenthesis `(`, an optional [link destination], an optional +[link title], and a right parenthesis `)`. +These four components may be separated by spaces, tabs, and up to one line +ending. +If both [link destination] and [link title] are present, they *must* be +separated by spaces, tabs, and up to one line ending. + +The link's text consists of the inlines contained +in the [link text] (excluding the enclosing square brackets). +The link's URI consists of the link destination, excluding enclosing +`<...>` if present, with backslash-escapes in effect as described +above. The link's title consists of the link title, excluding its +enclosing delimiters, with backslash-escapes in effect as described +above. + +Here is a simple inline link: + +```````````````````````````````` example +[link](/uri "title") +. +link +```````````````````````````````` + + +The title, the link text and even +the destination may be omitted: + +```````````````````````````````` example +[link](/uri) +. +link +```````````````````````````````` + +```````````````````````````````` example +[](./target.md) +. + +```````````````````````````````` + + +```````````````````````````````` example +[link]() +. +link +```````````````````````````````` + + +```````````````````````````````` example +[link](<>) +. +link +```````````````````````````````` + + +```````````````````````````````` example +[]() +. + +```````````````````````````````` + +The destination can only contain spaces if it is +enclosed in pointy brackets: + +```````````````````````````````` example +[link](/my uri) +. +[link](/my uri) +```````````````````````````````` + +```````````````````````````````` example +[link]() +. +link +```````````````````````````````` + +The destination cannot contain line endings, +even if enclosed in pointy brackets: + +```````````````````````````````` example +[link](foo +bar) +. +[link](foo +bar) +```````````````````````````````` + +```````````````````````````````` example +[link]() +. +[link]() +```````````````````````````````` + +The destination can contain `)` if it is enclosed +in pointy brackets: + +```````````````````````````````` example +[a]() +. +a +```````````````````````````````` + +Pointy brackets that enclose links must be unescaped: + +```````````````````````````````` example +[link]() +. +[link](<foo>) +```````````````````````````````` + +These are not links, because the opening pointy bracket +is not matched properly: + +```````````````````````````````` example +[a]( +[a](c) +. +[a](<b)c +[a](<b)c> +[a](c) +```````````````````````````````` + +Parentheses inside the link destination may be escaped: + +```````````````````````````````` example +[link]($foo$) +. +link +```````````````````````````````` + +Any number of parentheses are allowed without escaping, as long as they are +balanced: + +```````````````````````````````` example +[link](foo(and(bar))) +. +link +```````````````````````````````` + +However, if you have unbalanced parentheses, you need to escape or use the +`<...>` form: + +```````````````````````````````` example +[link](foo(and(bar)) +. +[link](foo(and(bar)) +```````````````````````````````` + + +```````````````````````````````` example +[link](foo$and\(bar$) +. +link +```````````````````````````````` + + +```````````````````````````````` example +[link]() +. +link +```````````````````````````````` + + +Parentheses and other symbols can also be escaped, as usual +in Markdown: + +```````````````````````````````` example +[link](foo\)\:) +. +link +```````````````````````````````` + + +A link can contain fragment identifiers and queries: + +```````````````````````````````` example +[link](#fragment) + +[link](http://example.com#fragment) + +[link](http://example.com?foo=3#frag) +. +link +link +link +```````````````````````````````` + + +Note that a backslash before a non-escapable character is +just a backslash: + +```````````````````````````````` example +[link](foo\bar) +. +link +```````````````````````````````` + + +URL-escaping should be left alone inside the destination, as all +URL-escaped characters are also valid URL characters. Entity and +numerical character references in the destination will be parsed +into the corresponding Unicode code points, as usual. These may +be optionally URL-escaped when written as HTML, but this spec +does not enforce any particular policy for rendering URLs in +HTML or other formats. Renderers may make different decisions +about how to escape or normalize URLs in the output. + +```````````````````````````````` example +[link](foo%20bä) +. +link +```````````````````````````````` + + +Note that, because titles can often be parsed as destinations, +if you try to omit the destination and keep the title, you'll +get unexpected results: + +```````````````````````````````` example +[link]("title") +. +link +```````````````````````````````` + + +Titles may be in single quotes, double quotes, or parentheses: + +```````````````````````````````` example +[link](/url "title") +[link](/url 'title') +[link](/url (title)) +. +link +link +link +```````````````````````````````` + + +Backslash escapes and entity and numeric character references +may be used in titles: + +```````````````````````````````` example +[link](/url "title \""") +. +link +```````````````````````````````` + + +Titles must be separated from the link using spaces, tabs, and up to one line +ending. +Other [Unicode whitespace] like non-breaking space doesn't work. + +```````````````````````````````` example +[link](/url "title") +. +link +```````````````````````````````` + + +Nested balanced quotes are not allowed without escaping: + +```````````````````````````````` example +[link](/url "title "and" title") +. +[link](/url "title "and" title") +```````````````````````````````` + + +But it is easy to work around this by using a different quote type: + +```````````````````````````````` example +[link](/url 'title "and" title') +. +link +```````````````````````````````` + + +(Note: `Markdown.pl` did allow double quotes inside a double-quoted +title, and its test suite included a test demonstrating this. +But it is hard to see a good rationale for the extra complexity this +brings, since there are already many ways---backslash escaping, +entity and numeric character references, or using a different +quote type for the enclosing title---to write titles containing +double quotes. `Markdown.pl`'s handling of titles has a number +of other strange features. For example, it allows single-quoted +titles in inline links, but not reference links. And, in +reference links but not inline links, it allows a title to begin +with `"` and end with `)`. `Markdown.pl` 1.0.1 even allows +titles with no closing quotation mark, though 1.0.2b8 does not. +It seems preferable to adopt a simple, rational rule that works +the same way in inline links and link reference definitions.) + +Spaces, tabs, and up to one line ending is allowed around the destination and +title: + +```````````````````````````````` example +[link]( /uri + "title" ) +. +link +```````````````````````````````` + + +But it is not allowed between the link text and the +following parenthesis: + +```````````````````````````````` example +[link] (/uri) +. +[link] (/uri) +```````````````````````````````` + + +The link text may contain balanced brackets, but not unbalanced ones, +unless they are escaped: + +```````````````````````````````` example +[link [foo [bar]]](/uri) +. +link [foo [bar]] +```````````````````````````````` + + +```````````````````````````````` example +[link] bar](/uri) +. +[link] bar](/uri) +```````````````````````````````` + + +```````````````````````````````` example +[link [bar](/uri) +. +[link bar +```````````````````````````````` + + +```````````````````````````````` example +[link \[bar](/uri) +. +link [bar +```````````````````````````````` + + +The link text may contain inline content: + +```````````````````````````````` example +[link *foo **bar** `#`*](/uri) +. +link foo bar # +```````````````````````````````` + + +```````````````````````````````` example +[![moon](moon.jpg)](/uri) +. + +```````````````````````````````` + + +However, links may not contain other links, at any level of nesting. + +```````````````````````````````` example +[foo [bar](/uri)](/uri) +. +[foo bar](/uri) +```````````````````````````````` + + +```````````````````````````````` example +[foo *[bar [baz](/uri)](/uri)*](/uri) +. +[foo [bar baz](/uri)](/uri) +```````````````````````````````` + + +```````````````````````````````` example +![[[foo](uri1)](uri2)](uri3) +. + +```````````````````````````````` + + +These cases illustrate the precedence of link text grouping over +emphasis grouping: + +```````````````````````````````` example +*[foo*](/uri) +. +*foo* +```````````````````````````````` + + +```````````````````````````````` example +[foo *bar](baz*) +. +foo *bar +```````````````````````````````` + + +Note that brackets that *aren't* part of links do not take +precedence: + +```````````````````````````````` example +*foo [bar* baz] +. +foo [bar baz] +```````````````````````````````` + + +These cases illustrate the precedence of HTML tags, code spans, +and autolinks over link grouping: + +```````````````````````````````` example +[foo +. +[foo +```````````````````````````````` + + +```````````````````````````````` example +[foo`](/uri)` +. +[foo](/uri) +```````````````````````````````` + + +```````````````````````````````` example +[foo +. +[foohttp://example.com/?search=](uri) +```````````````````````````````` + + +There are three kinds of [reference link](@)s: +[full](#full-reference-link), [collapsed](#collapsed-reference-link), +and [shortcut](#shortcut-reference-link). + +A [full reference link](@) +consists of a [link text] immediately followed by a [link label] +that [matches] a [link reference definition] elsewhere in the document. + +A [link label](@) begins with a left bracket (`[`) and ends +with the first right bracket (`]`) that is not backslash-escaped. +Between these brackets there must be at least one character that is not a space, +tab, or line ending. +Unescaped square bracket characters are not allowed inside the +opening and closing square brackets of [link labels]. A link +label can have at most 999 characters inside the square +brackets. + +One label [matches](@) +another just in case their normalized forms are equal. To normalize a +label, strip off the opening and closing brackets, +perform the *Unicode case fold*, strip leading and trailing +spaces, tabs, and line endings, and collapse consecutive internal +spaces, tabs, and line endings to a single space. If there are multiple +matching reference link definitions, the one that comes first in the +document is used. (It is desirable in such cases to emit a warning.) + +The link's URI and title are provided by the matching [link +reference definition]. + +Here is a simple example: + +```````````````````````````````` example +[foo][bar] + +[bar]: /url "title" +. +foo +```````````````````````````````` + + +The rules for the [link text] are the same as with +[inline links]. Thus: + +The link text may contain balanced brackets, but not unbalanced ones, +unless they are escaped: + +```````````````````````````````` example +[link [foo [bar]]][ref] + +[ref]: /uri +. +link [foo [bar]] +```````````````````````````````` + + +```````````````````````````````` example +[link \[bar][ref] + +[ref]: /uri +. +link [bar +```````````````````````````````` + + +The link text may contain inline content: + +```````````````````````````````` example +[link *foo **bar** `#`*][ref] + +[ref]: /uri +. +link foo bar # +```````````````````````````````` + + +```````````````````````````````` example +[![moon](moon.jpg)][ref] + +[ref]: /uri +. + +```````````````````````````````` + + +However, links may not contain other links, at any level of nesting. + +```````````````````````````````` example +[foo [bar](/uri)][ref] + +[ref]: /uri +. +[foo bar]ref +```````````````````````````````` + + +```````````````````````````````` example +[foo *bar [baz][ref]*][ref] + +[ref]: /uri +. +[foo bar baz]ref +```````````````````````````````` + + +(In the examples above, we have two [shortcut reference links] +instead of one [full reference link].) + +The following cases illustrate the precedence of link text grouping over +emphasis grouping: + +```````````````````````````````` example +*[foo*][ref] + +[ref]: /uri +. +*foo* +```````````````````````````````` + + +```````````````````````````````` example +[foo *bar][ref]* + +[ref]: /uri +. +foo *bar* +```````````````````````````````` + + +These cases illustrate the precedence of HTML tags, code spans, +and autolinks over link grouping: + +```````````````````````````````` example +[foo + +[ref]: /uri +. +[foo +```````````````````````````````` + + +```````````````````````````````` example +[foo`][ref]` + +[ref]: /uri +. +[foo][ref] +```````````````````````````````` + + +```````````````````````````````` example +[foo + +[ref]: /uri +. +[foohttp://example.com/?search=][ref] +```````````````````````````````` + + +Matching is case-insensitive: + +```````````````````````````````` example +[foo][BaR] + +[bar]: /url "title" +. +foo +```````````````````````````````` + + +Unicode case fold is used: + +```````````````````````````````` example +[ẞ] + +[SS]: /url +. +ẞ +```````````````````````````````` + + +Consecutive internal spaces, tabs, and line endings are treated as one space for +purposes of determining matching: + +```````````````````````````````` example +[Foo + bar]: /url + +[Baz][Foo bar] +. +Baz +```````````````````````````````` + + +No spaces, tabs, or line endings are allowed between the [link text] and the +[link label]: + +```````````````````````````````` example +[foo] [bar] + +[bar]: /url "title" +. +[foo] bar +```````````````````````````````` + + +```````````````````````````````` example +[foo] +[bar] + +[bar]: /url "title" +. +[foo] +bar +```````````````````````````````` + + +This is a departure from John Gruber's original Markdown syntax +description, which explicitly allows whitespace between the link +text and the link label. It brings reference links in line with +[inline links], which (according to both original Markdown and +this spec) cannot have whitespace after the link text. More +importantly, it prevents inadvertent capture of consecutive +[shortcut reference links]. If whitespace is allowed between the +link text and the link label, then in the following we will have +a single reference link, not two shortcut reference links, as +intended: + +``` markdown +[foo] +[bar] + +[foo]: /url1 +[bar]: /url2 +``` + +(Note that [shortcut reference links] were introduced by Gruber +himself in a beta version of `Markdown.pl`, but never included +in the official syntax description. Without shortcut reference +links, it is harmless to allow space between the link text and +link label; but once shortcut references are introduced, it is +too dangerous to allow this, as it frequently leads to +unintended results.) + +When there are multiple matching [link reference definitions], +the first is used: + +```````````````````````````````` example +[foo]: /url1 + +[foo]: /url2 + +[bar][foo] +. +bar +```````````````````````````````` + + +Note that matching is performed on normalized strings, not parsed +inline content. So the following does not match, even though the +labels define equivalent inline content: + +```````````````````````````````` example +[bar][foo\!] + +[foo!]: /url +. +[bar][foo!] +```````````````````````````````` + + +[Link labels] cannot contain brackets, unless they are +backslash-escaped: + +```````````````````````````````` example +[foo][ref[] + +[ref[]: /uri +. +[foo][ref[] +[ref[]: /uri +```````````````````````````````` + + +```````````````````````````````` example +[foo][ref[bar]] + +[ref[bar]]: /uri +. +[foo][ref[bar]] +[ref[bar]]: /uri +```````````````````````````````` + + +```````````````````````````````` example +[[[foo]]] + +[[[foo]]]: /url +. +[[[foo]]] +[[[foo]]]: /url +```````````````````````````````` + + +```````````````````````````````` example +[foo][ref\[] + +[ref\[]: /uri +. +foo +```````````````````````````````` + + +Note that in this example `]` is not backslash-escaped: + +```````````````````````````````` example +[bar\\]: /uri + +[bar\\] +. +bar\ +```````````````````````````````` + + +A [link label] must contain at least one character that is not a space, tab, or +line ending: + +```````````````````````````````` example +[] + +[]: /uri +. +[] +[]: /uri +```````````````````````````````` + + +```````````````````````````````` example +[ + ] + +[ + ]: /uri +. +[ +] +[ +]: /uri +```````````````````````````````` + + +A [collapsed reference link](@) +consists of a [link label] that [matches] a +[link reference definition] elsewhere in the +document, followed by the string `[]`. +The contents of the first link label are parsed as inlines, +which are used as the link's text. The link's URI and title are +provided by the matching reference link definition. Thus, +`[foo][]` is equivalent to `[foo][foo]`. + +```````````````````````````````` example +[foo][] + +[foo]: /url "title" +. +foo +```````````````````````````````` + + +```````````````````````````````` example +[*foo* bar][] + +[*foo* bar]: /url "title" +. +foo bar +```````````````````````````````` + + +The link labels are case-insensitive: + +```````````````````````````````` example +[Foo][] + +[foo]: /url "title" +. +Foo +```````````````````````````````` + + + +As with full reference links, spaces, tabs, or line endings are not +allowed between the two sets of brackets: + +```````````````````````````````` example +[foo] +[] + +[foo]: /url "title" +. +foo +[] +```````````````````````````````` + + +A [shortcut reference link](@) +consists of a [link label] that [matches] a +[link reference definition] elsewhere in the +document and is not followed by `[]` or a link label. +The contents of the first link label are parsed as inlines, +which are used as the link's text. The link's URI and title +are provided by the matching link reference definition. +Thus, `[foo]` is equivalent to `[foo][]`. + +```````````````````````````````` example +[foo] + +[foo]: /url "title" +. +foo +```````````````````````````````` + + +```````````````````````````````` example +[*foo* bar] + +[*foo* bar]: /url "title" +. +foo bar +```````````````````````````````` + + +```````````````````````````````` example +[[*foo* bar]] + +[*foo* bar]: /url "title" +. +[foo bar] +```````````````````````````````` + + +```````````````````````````````` example +[[bar [foo] + +[foo]: /url +. +[[bar foo +```````````````````````````````` + + +The link labels are case-insensitive: + +```````````````````````````````` example +[Foo] + +[foo]: /url "title" +. +Foo +```````````````````````````````` + + +A space after the link text should be preserved: + +```````````````````````````````` example +[foo] bar + +[foo]: /url +. +foo bar +```````````````````````````````` + + +If you just want bracketed text, you can backslash-escape the +opening bracket to avoid links: + +```````````````````````````````` example +\[foo] + +[foo]: /url "title" +. +[foo] +```````````````````````````````` + + +Note that this is a link, because a link label ends with the first +following closing bracket: + +```````````````````````````````` example +[foo*]: /url + +*[foo*] +. +*foo* +```````````````````````````````` + + +Full and compact references take precedence over shortcut +references: + +```````````````````````````````` example +[foo][bar] + +[foo]: /url1 +[bar]: /url2 +. +foo +```````````````````````````````` + +```````````````````````````````` example +[foo][] + +[foo]: /url1 +. +foo +```````````````````````````````` + +Inline links also take precedence: + +```````````````````````````````` example +[foo]() + +[foo]: /url1 +. +foo +```````````````````````````````` + +```````````````````````````````` example +[foo](not a link) + +[foo]: /url1 +. +foo(not a link) +```````````````````````````````` + +In the following case `[bar][baz]` is parsed as a reference, +`[foo]` as normal text: + +```````````````````````````````` example +[foo][bar][baz] + +[baz]: /url +. +[foo]bar +```````````````````````````````` + + +Here, though, `[foo][bar]` is parsed as a reference, since +`[bar]` is defined: + +```````````````````````````````` example +[foo][bar][baz] + +[baz]: /url1 +[bar]: /url2 +. +foobaz +```````````````````````````````` + + +Here `[foo]` is not parsed as a shortcut reference, because it +is followed by a link label (even though `[bar]` is not defined): + +```````````````````````````````` example +[foo][bar][baz] + +[baz]: /url1 +[foo]: /url2 +. +[foo]bar +```````````````````````````````` + + + +## Images + +Syntax for images is like the syntax for links, with one +difference. Instead of [link text], we have an +[image description](@). The rules for this are the +same as for [link text], except that (a) an +image description starts with `![` rather than `[`, and +(b) an image description may contain links. +An image description has inline elements +as its contents. When an image is rendered to HTML, +this is standardly used as the image's `alt` attribute. + +```````````````````````````````` example +![foo](/url "title") +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo *bar*] + +[foo *bar*]: train.jpg "train & tracks" +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo ![bar](/url)](/url2) +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo [bar](/url)](/url2) +. + +```````````````````````````````` + + +Though this spec is concerned with parsing, not rendering, it is +recommended that in rendering to HTML, only the plain string content +of the [image description] be used. Note that in +the above example, the alt attribute's value is `foo bar`, not `foo +[bar](/url)` or `foo bar`. Only the plain string +content is rendered, without formatting. + +```````````````````````````````` example +![foo *bar*][] + +[foo *bar*]: train.jpg "train & tracks" +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo *bar*][foobar] + +[FOOBAR]: train.jpg "train & tracks" +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo](train.jpg) +. + +```````````````````````````````` + + +```````````````````````````````` example +My ![foo bar](/path/to/train.jpg "title" ) +. +My +```````````````````````````````` + + +```````````````````````````````` example +![foo]() +. + +```````````````````````````````` + + +```````````````````````````````` example +![](/url) +. + +```````````````````````````````` + + +Reference-style: + +```````````````````````````````` example +![foo][bar] + +[bar]: /url +. + +```````````````````````````````` + + +```````````````````````````````` example +![foo][bar] + +[BAR]: /url +. + +```````````````````````````````` + + +Collapsed: + +```````````````````````````````` example +![foo][] + +[foo]: /url "title" +. + +```````````````````````````````` + + +```````````````````````````````` example +![*foo* bar][] + +[*foo* bar]: /url "title" +. + +```````````````````````````````` + + +The labels are case-insensitive: + +```````````````````````````````` example +![Foo][] + +[foo]: /url "title" +. + +```````````````````````````````` + + +As with reference links, spaces, tabs, and line endings, are not allowed +between the two sets of brackets: + +```````````````````````````````` example +![foo] +[] + +[foo]: /url "title" +. + +[] +```````````````````````````````` + + +Shortcut: + +```````````````````````````````` example +![foo] + +[foo]: /url "title" +. + +```````````````````````````````` + + +```````````````````````````````` example +![*foo* bar] + +[*foo* bar]: /url "title" +. + +```````````````````````````````` + + +Note that link labels cannot contain unescaped brackets: + +```````````````````````````````` example +![[foo]] + +[[foo]]: /url "title" +. +![[foo]] +[[foo]]: /url "title" +```````````````````````````````` + + +The link labels are case-insensitive: + +```````````````````````````````` example +![Foo] + +[foo]: /url "title" +. + +```````````````````````````````` + + +If you just want a literal `!` followed by bracketed text, you can +backslash-escape the opening `[`: + +```````````````````````````````` example +!\[foo] + +[foo]: /url "title" +. +![foo] +```````````````````````````````` + + +If you want a link after a literal `!`, backslash-escape the +`!`: + +```````````````````````````````` example +\![foo] + +[foo]: /url "title" +. +!foo +```````````````````````````````` + + +## Autolinks + +[Autolink](@)s are absolute URIs and email addresses inside +`<` and `>`. They are parsed as links, with the URL or email address +as the link label. + +A [URI autolink](@) consists of `<`, followed by an +[absolute URI] followed by `>`. It is parsed as +a link to the URI, with the URI as the link's label. + +An [absolute URI](@), +for these purposes, consists of a [scheme] followed by a colon (`:`) +followed by zero or more characters other [ASCII control +characters][ASCII control character], [space], `<`, and `>`. +If the URI includes these characters, they must be percent-encoded +(e.g. `%20` for a space). + +For purposes of this spec, a [scheme](@) is any sequence +of 2--32 characters beginning with an ASCII letter and followed +by any combination of ASCII letters, digits, or the symbols plus +("+"), period ("."), or hyphen ("-"). + +Here are some valid autolinks: + +```````````````````````````````` example + +. +http://foo.bar.baz +```````````````````````````````` + + +```````````````````````````````` example + +. +http://foo.bar.baz/test?q=hello&id=22&boolean +```````````````````````````````` + + +```````````````````````````````` example + +. +irc://foo.bar:2233/baz +```````````````````````````````` + + +Uppercase is also fine: + +```````````````````````````````` example + +. +MAILTO:FOO@BAR.BAZ +```````````````````````````````` + + +Note that many strings that count as [absolute URIs] for +purposes of this spec are not valid URIs, because their +schemes are not registered or because of other problems +with their syntax: + +```````````````````````````````` example + +. +a+b+c:d +```````````````````````````````` + + +```````````````````````````````` example + +. +made-up-scheme://foo,bar +```````````````````````````````` + + +```````````````````````````````` example + +. +http://../ +```````````````````````````````` + + +```````````````````````````````` example + +. +localhost:5001/foo +```````````````````````````````` + + +Spaces are not allowed in autolinks: + +```````````````````````````````` example + +. +<http://foo.bar/baz bim> +```````````````````````````````` + + +Backslash-escapes do not work inside autolinks: + +```````````````````````````````` example + +. +http://example.com/\[\ +```````````````````````````````` + + +An [email autolink](@) +consists of `<`, followed by an [email address], +followed by `>`. The link's label is the email address, +and the URL is `mailto:` followed by the email address. + +An [email address](@), +for these purposes, is anything that matches +the [non-normative regex from the HTML5 +spec](https://html.spec.whatwg.org/multipage/forms.html#e-mail-state-(type=email)): + + /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])? + (?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/ + +Examples of email autolinks: + +```````````````````````````````` example + +. +foo@bar.example.com +```````````````````````````````` + + +```````````````````````````````` example + +. +foo+special@Bar.baz-bar0.com +```````````````````````````````` + + +Backslash-escapes do not work inside email autolinks: + +```````````````````````````````` example + +. +<foo+@bar.example.com> +```````````````````````````````` + + +These are not autolinks: + +```````````````````````````````` example +<> +. +<> +```````````````````````````````` + + +```````````````````````````````` example +< http://foo.bar > +. +< http://foo.bar > +```````````````````````````````` + + +```````````````````````````````` example + +. +<m:abc> +```````````````````````````````` + + +```````````````````````````````` example + +. +<foo.bar.baz> +```````````````````````````````` + + +```````````````````````````````` example +http://example.com +. +http://example.com +```````````````````````````````` + + +```````````````````````````````` example +foo@bar.example.com +. +foo@bar.example.com +```````````````````````````````` + + +## Raw HTML + +Text between `<` and `>` that looks like an HTML tag is parsed as a +raw HTML tag and will be rendered in HTML without escaping. +Tag and attribute names are not limited to current HTML tags, +so custom tags (and even, say, DocBook tags) may be used. + +Here is the grammar for tags: + +A [tag name](@) consists of an ASCII letter +followed by zero or more ASCII letters, digits, or +hyphens (`-`). + +An [attribute](@) consists of spaces, tabs, and up to one line ending, +an [attribute name], and an optional +[attribute value specification]. + +An [attribute name](@) +consists of an ASCII letter, `_`, or `:`, followed by zero or more ASCII +letters, digits, `_`, `.`, `:`, or `-`. (Note: This is the XML +specification restricted to ASCII. HTML5 is laxer.) + +An [attribute value specification](@) +consists of optional spaces, tabs, and up to one line ending, +a `=` character, optional spaces, tabs, and up to one line ending, +and an [attribute value]. + +An [attribute value](@) +consists of an [unquoted attribute value], +a [single-quoted attribute value], or a [double-quoted attribute value]. + +An [unquoted attribute value](@) +is a nonempty string of characters not +including spaces, tabs, line endings, `"`, `'`, `=`, `<`, `>`, or `` ` ``. + +A [single-quoted attribute value](@) +consists of `'`, zero or more +characters not including `'`, and a final `'`. + +A [double-quoted attribute value](@) +consists of `"`, zero or more +characters not including `"`, and a final `"`. + +An [open tag](@) consists of a `<` character, a [tag name], +zero or more [attributes], optional spaces, tabs, and up to one line ending, +an optional `/` character, and a `>` character. + +A [closing tag](@) consists of the string ``. + +An [HTML comment](@) consists of ``, +where *text* does not start with `>` or `->`, does not end with `-`, +and does not contain `--`. (See the +[HTML5 spec](http://www.w3.org/TR/html5/syntax.html#comments).) + +A [processing instruction](@) +consists of the string ``, and the string +`?>`. + +A [declaration](@) consists of the string ``, and the character `>`. + +A [CDATA section](@) consists of +the string ``, and the string `]]>`. + +An [HTML tag](@) consists of an [open tag], a [closing tag], +an [HTML comment], a [processing instruction], a [declaration], +or a [CDATA section]. + +Here are some simple open tags: + +```````````````````````````````` example + +. + +```````````````````````````````` + + +Empty elements: + +```````````````````````````````` example + +. + +```````````````````````````````` + + +Whitespace is allowed: + +```````````````````````````````` example + +. + +```````````````````````````````` + + +With attributes: + +```````````````````````````````` example + +. + +```````````````````````````````` + + +Custom tag names can be used: + +```````````````````````````````` example +Foo +. +Foo +```````````````````````````````` + + +Illegal tag names, not parsed as HTML: + +```````````````````````````````` example +<33> <__> +. +<33> <__> +```````````````````````````````` + + +Illegal attribute names: + +```````````````````````````````` example + +. +<a h*#ref="hi"> +```````````````````````````````` + + +Illegal attribute values: + +```````````````````````````````` example + +. +</a href="foo"> +```````````````````````````````` + + +Comments: + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +```````````````````````````````` example +foo +. +foo  +```````````````````````````````` + + +Not comments: + +```````````````````````````````` example +foo foo --> + +foo +. +foo  +foo  +```````````````````````````````` + + +Processing instructions: + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +Declarations: + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +CDATA sections: + +```````````````````````````````` example +foo &<]]> +. +foo &<]]> +```````````````````````````````` + + +Entity and numeric character references are preserved in HTML +attributes: + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +Backslash escapes do not work in HTML attributes: + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +```````````````````````````````` example + +. +<a href="""> +```````````````````````````````` + + +## Hard line breaks + +A line ending (not in a code span or HTML tag) that is preceded +by two or more spaces and does not occur at the end of a block +is parsed as a [hard line break](@) (rendered +in HTML as a ` ` tag): + +```````````````````````````````` example +foo +baz +. +foo +baz +```````````````````````````````` + + +For a more visible alternative, a backslash before the +[line ending] may be used instead of two or more spaces: + +```````````````````````````````` example +foo\ +baz +. +foo +baz +```````````````````````````````` + + +More than two spaces can be used: + +```````````````````````````````` example +foo +baz +. +foo +baz +```````````````````````````````` + + +Leading spaces at the beginning of the next line are ignored: + +```````````````````````````````` example +foo + bar +. +foo +bar +```````````````````````````````` + + +```````````````````````````````` example +foo\ + bar +. +foo +bar +```````````````````````````````` + + +Hard line breaks can occur inside emphasis, links, and other constructs +that allow inline content: + +```````````````````````````````` example +*foo +bar* +. +foo +bar +```````````````````````````````` + + +```````````````````````````````` example +*foo\ +bar* +. +foo +bar +```````````````````````````````` + + +Hard line breaks do not occur inside code spans + +```````````````````````````````` example +`code +span` +. +code span +```````````````````````````````` + + +```````````````````````````````` example +`code\ +span` +. +code\ span +```````````````````````````````` + + +or HTML tags: + +```````````````````````````````` example + +. + +```````````````````````````````` + + +```````````````````````````````` example + +. + +```````````````````````````````` + + +Hard line breaks are for separating inline content within a block. +Neither syntax for hard line breaks works at the end of a paragraph or +other block element: + +```````````````````````````````` example +foo\ +. +foo\ +```````````````````````````````` + + +```````````````````````````````` example +foo +. +foo +```````````````````````````````` + + +```````````````````````````````` example +### foo\ +. +foo\ +```````````````````````````````` + + +```````````````````````````````` example +### foo +. +foo +```````````````````````````````` + + +## Soft line breaks + +A regular line ending (not in a code span or HTML tag) that is not +preceded by two or more spaces or a backslash is parsed as a +[softbreak](@). (A soft line break may be rendered in HTML either as a +[line ending] or as a space. The result will be the same in +browsers. In the examples here, a [line ending] will be used.) + +```````````````````````````````` example +foo +baz +. +foo +baz +```````````````````````````````` + + +Spaces at the end of the line and beginning of the next line are +removed: + +```````````````````````````````` example +foo + baz +. +foo +baz +```````````````````````````````` + + +A conforming parser may render a soft line break in HTML either as a +line ending or as a space. + +A renderer may also provide an option to render soft line breaks +as hard line breaks. + +## Textual content + +Any characters not given an interpretation by the above rules will +be parsed as plain textual content. + +```````````````````````````````` example +hello $.;'there +. +hello $.;'there +```````````````````````````````` + + +```````````````````````````````` example +Foo χρῆν +. +Foo χρῆν +```````````````````````````````` + + +Internal spaces are preserved verbatim: + +```````````````````````````````` example +Multiple spaces +. +Multiple spaces +```````````````````````````````` + + + + +# Appendix: A parsing strategy + +In this appendix we describe some features of the parsing strategy +used in the CommonMark reference implementations. + +## Overview + +Parsing has two phases: + +1. In the first phase, lines of input are consumed and the block +structure of the document---its division into paragraphs, block quotes, +list items, and so on---is constructed. Text is assigned to these +blocks but not parsed. Link reference definitions are parsed and a +map of links is constructed. + +2. In the second phase, the raw text contents of paragraphs and headings +are parsed into sequences of Markdown inline elements (strings, +code spans, links, emphasis, and so on), using the map of link +references constructed in phase 1. + +At each point in processing, the document is represented as a tree of +**blocks**. The root of the tree is a `document` block. The `document` +may have any number of other blocks as **children**. These children +may, in turn, have other blocks as children. The last child of a block +is normally considered **open**, meaning that subsequent lines of input +can alter its contents. (Blocks that are not open are **closed**.) +Here, for example, is a possible document tree, with the open blocks +marked by arrows: + +``` tree +-> document + -> block_quote + paragraph + "Lorem ipsum dolor\nsit amet." + -> list (type=bullet tight=true bullet_char=-) + list_item + paragraph + "Qui *quodsi iracundia*" + -> list_item + -> paragraph + "aliquando id" +``` + +## Phase 1: block structure + +Each line that is processed has an effect on this tree. The line is +analyzed and, depending on its contents, the document may be altered +in one or more of the following ways: + +1. One or more open blocks may be closed. +2. One or more new blocks may be created as children of the + last open block. +3. Text may be added to the last (deepest) open block remaining + on the tree. + +Once a line has been incorporated into the tree in this way, +it can be discarded, so input can be read in a stream. + +For each line, we follow this procedure: + +1. First we iterate through the open blocks, starting with the +root document, and descending through last children down to the last +open block. Each block imposes a condition that the line must satisfy +if the block is to remain open. For example, a block quote requires a +`>` character. A paragraph requires a non-blank line. +In this phase we may match all or just some of the open +blocks. But we cannot close unmatched blocks yet, because we may have a +[lazy continuation line]. + +2. Next, after consuming the continuation markers for existing +blocks, we look for new block starts (e.g. `>` for a block quote). +If we encounter a new block start, we close any blocks unmatched +in step 1 before creating the new block as a child of the last +matched container block. + +3. Finally, we look at the remainder of the line (after block +markers like `>`, list markers, and indentation have been consumed). +This is text that can be incorporated into the last open +block (a paragraph, code block, heading, or raw HTML). + +Setext headings are formed when we see a line of a paragraph +that is a [setext heading underline]. + +Reference link definitions are detected when a paragraph is closed; +the accumulated text lines are parsed to see if they begin with +one or more reference link definitions. Any remainder becomes a +normal paragraph. + +We can see how this works by considering how the tree above is +generated by four lines of Markdown: + +``` markdown +> Lorem ipsum dolor +sit amet. +> - Qui *quodsi iracundia* +> - aliquando id +``` + +At the outset, our document model is just + +``` tree +-> document +``` + +The first line of our text, + +``` markdown +> Lorem ipsum dolor +``` + +causes a `block_quote` block to be created as a child of our +open `document` block, and a `paragraph` block as a child of +the `block_quote`. Then the text is added to the last open +block, the `paragraph`: + +``` tree +-> document + -> block_quote + -> paragraph + "Lorem ipsum dolor" +``` + +The next line, + +``` markdown +sit amet. +``` + +is a "lazy continuation" of the open `paragraph`, so it gets added +to the paragraph's text: + +``` tree +-> document + -> block_quote + -> paragraph + "Lorem ipsum dolor\nsit amet." +``` + +The third line, + +``` markdown +> - Qui *quodsi iracundia* +``` + +causes the `paragraph` block to be closed, and a new `list` block +opened as a child of the `block_quote`. A `list_item` is also +added as a child of the `list`, and a `paragraph` as a child of +the `list_item`. The text is then added to the new `paragraph`: + +``` tree +-> document + -> block_quote + paragraph + "Lorem ipsum dolor\nsit amet." + -> list (type=bullet tight=true bullet_char=-) + -> list_item + -> paragraph + "Qui *quodsi iracundia*" +``` + +The fourth line, + +``` markdown +> - aliquando id +``` + +causes the `list_item` (and its child the `paragraph`) to be closed, +and a new `list_item` opened up as child of the `list`. A `paragraph` +is added as a child of the new `list_item`, to contain the text. +We thus obtain the final tree: + +``` tree +-> document + -> block_quote + paragraph + "Lorem ipsum dolor\nsit amet." + -> list (type=bullet tight=true bullet_char=-) + list_item + paragraph + "Qui *quodsi iracundia*" + -> list_item + -> paragraph + "aliquando id" +``` + +## Phase 2: inline structure + +Once all of the input has been parsed, all open blocks are closed. + +We then "walk the tree," visiting every node, and parse raw +string contents of paragraphs and headings as inlines. At this +point we have seen all the link reference definitions, so we can +resolve reference links as we go. + +``` tree +document + block_quote + paragraph + str "Lorem ipsum dolor" + softbreak + str "sit amet." + list (type=bullet tight=true bullet_char=-) + list_item + paragraph + str "Qui " + emph + str "quodsi iracundia" + list_item + paragraph + str "aliquando id" +``` + +Notice how the [line ending] in the first paragraph has +been parsed as a `softbreak`, and the asterisks in the first list item +have become an `emph`. + +### An algorithm for parsing nested emphasis and links + +By far the trickiest part of inline parsing is handling emphasis, +strong emphasis, links, and images. This is done using the following +algorithm. + +When we're parsing inlines and we hit either + +- a run of `*` or `_` characters, or +- a `[` or `![` + +we insert a text node with these symbols as its literal content, and we +add a pointer to this text node to the [delimiter stack](@). + +The [delimiter stack] is a doubly linked list. Each +element contains a pointer to a text node, plus information about + +- the type of delimiter (`[`, `![`, `*`, `_`) +- the number of delimiters, +- whether the delimiter is "active" (all are active to start), and +- whether the delimiter is a potential opener, a potential closer, + or both (which depends on what sort of characters precede + and follow the delimiters). + +When we hit a `]` character, we call the *look for link or image* +procedure (see below). + +When we hit the end of the input, we call the *process emphasis* +procedure (see below), with `stack_bottom` = NULL. + +#### *look for link or image* + +Starting at the top of the delimiter stack, we look backwards +through the stack for an opening `[` or `![` delimiter. + +- If we don't find one, we return a literal text node `]`. + +- If we do find one, but it's not *active*, we remove the inactive + delimiter from the stack, and return a literal text node `]`. + +- If we find one and it's active, then we parse ahead to see if + we have an inline link/image, reference link/image, compact reference + link/image, or shortcut reference link/image. + + + If we don't, then we remove the opening delimiter from the + delimiter stack and return a literal text node `]`. + + + If we do, then + + * We return a link or image node whose children are the inlines + after the text node pointed to by the opening delimiter. + + * We run *process emphasis* on these inlines, with the `[` opener + as `stack_bottom`. + + * We remove the opening delimiter. + + * If we have a link (and not an image), we also set all + `[` delimiters before the opening delimiter to *inactive*. (This + will prevent us from getting links within links.) + +#### *process emphasis* + +Parameter `stack_bottom` sets a lower bound to how far we +descend in the [delimiter stack]. If it is NULL, we can +go all the way to the bottom. Otherwise, we stop before +visiting `stack_bottom`. + +Let `current_position` point to the element on the [delimiter stack] +just above `stack_bottom` (or the first element if `stack_bottom` +is NULL). + +We keep track of the `openers_bottom` for each delimiter +type (`*`, `_`). Initialize this to `stack_bottom`. + +Then we repeat the following until we run out of potential +closers: + +- Move `current_position` forward in the delimiter stack (if needed) + until we find the first potential closer with delimiter `*` or `_`. + (This will be the potential closer closest + to the beginning of the input -- the first one in parse order.) + +- Now, look back in the stack (staying above `stack_bottom` and + the `openers_bottom` for this delimiter type) for the + first matching potential opener ("matching" means same delimiter). + +- If one is found: + + + Figure out whether we have emphasis or strong emphasis: + if both closer and opener spans have length >= 2, we have + strong, otherwise regular. + + + Insert an emph or strong emph node accordingly, after + the text node corresponding to the opener. + + + Remove any delimiters between the opener and closer from + the delimiter stack. + + + Remove 1 (for regular emph) or 2 (for strong emph) delimiters + from the opening and closing text nodes. If they become empty + as a result, remove them and remove the corresponding element + of the delimiter stack. If the closing node is removed, reset + `current_position` to the next element in the stack. + +- If none is found: + + + Set `openers_bottom` for this delimiter type to the element + before the first potential opener that was rejected solely + because of the "multiple of 3" rule, or to the element + before `current_position`, if there are no such potential + openers. (We know that there are no openers for this kind + of closer up to and including this point, so this puts a + lower bound on future searches.) + + + If the closer at `current_position` is not a potential opener, + remove it from the delimiter stack (since we know it can't + be a closer either). + + + Advance `current_position` to the next element in the stack. + +After we're done, we remove all delimiters above `stack_bottom` from the +delimiter stack. diff --git a/flexmark-test-suite/flexmark-test-suite.iml b/flexmark-test-suite/flexmark-test-suite.iml index f43dac55f..4bc8a456a 100644 --- a/flexmark-test-suite/flexmark-test-suite.iml +++ b/flexmark-test-suite/flexmark-test-suite.iml @@ -9,7 +9,7 @@ - + @@ -35,6 +35,7 @@ + @@ -59,4 +60,4 @@ - \ No newline at end of file + diff --git a/flexmark-test-suite/src/test/java/com/vladsch/flexmark/test/suite/FlexmarkTestSuiteNoDocx.java b/flexmark-test-suite/src/test/java/com/vladsch/flexmark/test/suite/FlexmarkTestSuiteNoDocx.java index 9e2f1a14c..d1f5bdc03 100644 --- a/flexmark-test-suite/src/test/java/com/vladsch/flexmark/test/suite/FlexmarkTestSuiteNoDocx.java +++ b/flexmark-test-suite/src/test/java/com/vladsch/flexmark/test/suite/FlexmarkTestSuiteNoDocx.java @@ -25,6 +25,7 @@ import com.vladsch.flexmark.ext.jekyll.tag.ExtJekyllTagTestSuite; import com.vladsch.flexmark.ext.macros.ExtMacrosTestSuite; import com.vladsch.flexmark.ext.media.tags.ExtMediaTagsTestSuite; +import com.vladsch.flexmark.ext.resizable.image.ExtResizableImageTestSuite; import com.vladsch.flexmark.ext.spec.example.ExtSpecExampleTestSuite; import com.vladsch.flexmark.ext.superscript.ExtSuperscriptTestSuite; import com.vladsch.flexmark.ext.tables.ExtTablesTestSuite; @@ -73,6 +74,7 @@ ExtJekyllTagTestSuite.class, ExtMacrosTestSuite.class, ExtMediaTagsTestSuite.class, + ExtResizableImageTestSuite.class, ExtSpecExampleTestSuite.class, ExtSuperscriptTestSuite.class, ExtTablesTestSuite.class, diff --git a/flexmark-test-util/flexmark-test-util.iml b/flexmark-test-util/flexmark-test-util.iml index a0731c590..f9b578719 100644 --- a/flexmark-test-util/flexmark-test-util.iml +++ b/flexmark-test-util/flexmark-test-util.iml @@ -16,8 +16,8 @@ - + - + \ No newline at end of file diff --git a/flexmark-test-util/pom.xml b/flexmark-test-util/pom.xml index 71c2d99cf..4b6c75f1a 100644 --- a/flexmark-test-util/pom.xml +++ b/flexmark-test-util/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-test-util @@ -39,7 +39,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/ExceptionMatcher.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/ExceptionMatcher.java index 42623c740..0b0d25f4b 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/ExceptionMatcher.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/ExceptionMatcher.java @@ -25,8 +25,13 @@ public boolean matches(Object o) { } } else if (o instanceof Throwable) { if (o.toString().startsWith(prefix)) { + String input; Throwable throwable = (Throwable) o; - String input = throwable.getCause().toString(); + if (throwable.getCause() == null) { + input = o.toString(); + } else { + input = throwable.getCause().toString(); + } return pattern.matcher(input).matches(); } } diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/FlexmarkResourceUrlResolver.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/FlexmarkResourceUrlResolver.java index b975d8632..3232d2f41 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/FlexmarkResourceUrlResolver.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/FlexmarkResourceUrlResolver.java @@ -3,27 +3,46 @@ import com.vladsch.flexmark.test.util.spec.ResourceResolverManager; import com.vladsch.flexmark.test.util.spec.ResourceUrlResolver; -import static com.vladsch.flexmark.test.util.spec.ResourceUrlResolver.isFileProtocol; -import static com.vladsch.flexmark.test.util.spec.ResourceUrlResolver.removeProtocol; - public class FlexmarkResourceUrlResolver { public static void registerUrlResolvers() { - ResourceResolverManager.registerUrlResolver(new TargetTestClassUrlResolver()); + ResourceResolverManager.registerUrlResolver(new TargetTestResourceUrlResolver()); + ResourceResolverManager.registerUrlResolver(new BuildTestResourceUrlResolver()); ResourceResolverManager.registerUrlResolver(new OutTestResourcesUrlResolver()); } - private static class TargetTestClassUrlResolver implements ResourceUrlResolver { - static final String TARGET_TEST_CLASSES = "/target/test-classes/"; + private static class TargetTestResourceUrlResolver implements ResourceUrlResolver { + static final String TEST_RESOURCES = "/target/test-classes/"; + public static final String SRC_TEST_RESOURCES = "/src/test/resources/"; + + TargetTestResourceUrlResolver() { } + + @Override + public String apply(String externalForm) { + if (ResourceUrlResolver.isFileProtocol(externalForm)) { + String noFileProtocol = ResourceUrlResolver.removeProtocol(externalForm); + + if (noFileProtocol.contains(TEST_RESOURCES)) { + return noFileProtocol.replaceFirst(TEST_RESOURCES, SRC_TEST_RESOURCES); + } + } + + return null; + } + } + + private static class BuildTestResourceUrlResolver implements ResourceUrlResolver { + static final String TEST_RESOURCES = "/build/resources/test/"; + public static final String SRC_TEST_RESOURCES = "/src/test/resources/"; - TargetTestClassUrlResolver() {} + BuildTestResourceUrlResolver() { } @Override public String apply(String externalForm) { - if (isFileProtocol(externalForm)) { - String noFileProtocol = removeProtocol(externalForm); + if (ResourceUrlResolver.isFileProtocol(externalForm)) { + String noFileProtocol = ResourceUrlResolver.removeProtocol(externalForm); - if (noFileProtocol.contains(TARGET_TEST_CLASSES)) { - return noFileProtocol.replace(TARGET_TEST_CLASSES, "/src/test/resources/"); + if (noFileProtocol.contains(TEST_RESOURCES)) { + return noFileProtocol.replaceFirst(TEST_RESOURCES, SRC_TEST_RESOURCES); } } @@ -33,19 +52,20 @@ public String apply(String externalForm) { private static class OutTestResourcesUrlResolver implements ResourceUrlResolver { static final String OUT_TEST = "/out/test/"; + public static final String SRC_TEST_RESOURCES = "/src/test/resources/"; - OutTestResourcesUrlResolver() {} + OutTestResourcesUrlResolver() { } @Override public String apply(String externalForm) { - if (isFileProtocol(externalForm)) { - String noFileProtocol = removeProtocol(externalForm); + if (ResourceUrlResolver.isFileProtocol(externalForm)) { + String noFileProtocol = ResourceUrlResolver.removeProtocol(externalForm); int pos = noFileProtocol.indexOf(OUT_TEST); if (pos > 0) { int pathPos = noFileProtocol.indexOf("/", pos + OUT_TEST.length()); if (pathPos > 0) { - return noFileProtocol.substring(0, pos) + "/" + noFileProtocol.substring(pos + OUT_TEST.length(), pathPos) + "/src/test/resources/" + noFileProtocol.substring(pathPos + 1); + return noFileProtocol.substring(0, pos) + "/" + noFileProtocol.substring(pos + OUT_TEST.length(), pathPos) + SRC_TEST_RESOURCES + noFileProtocol.substring(pathPos + 1); } } } diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/IParseBase.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/IParseBase.java index 25d82f693..bff1a3bc6 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/IParseBase.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/IParseBase.java @@ -47,8 +47,8 @@ public boolean transferReferences(@NotNull Document document, @NotNull Document while (true) { int charsRead = bufferedReader.read(buffer); + if (charsRead < 0) break; file.append(buffer, 0, charsRead); - if (charsRead < buffer.length) break; } BasedSequence source = BasedSequence.of(file.toString()); diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceResolverManager.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceResolverManager.java index 5fc2e792e..5d3a7dd0c 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceResolverManager.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceResolverManager.java @@ -11,8 +11,12 @@ import static com.vladsch.flexmark.test.util.spec.ResourceUrlResolver.hasProtocol; public class ResourceResolverManager { + /** + * urlResolvers map test resource location url to source resource url to allow tests to + * output file URLs which refer to source location, not copies in test location + */ final private static ArrayList> urlResolvers = new ArrayList<>(); - + public static void registerUrlResolver(@NotNull Function resolver) { ResourceResolverManager.urlResolvers.add(resolver); } diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceUrlResolver.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceUrlResolver.java index b7a11f1f5..b9885a044 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceUrlResolver.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/ResourceUrlResolver.java @@ -4,6 +4,9 @@ import java.util.function.Function; +/** + * Used to resolve test resource URL from copy in test location to URL of source file + */ public interface ResourceUrlResolver extends Function { @Override String apply(String externalForm); diff --git a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/SpecReader.java b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/SpecReader.java index 76a8abfe0..18047998b 100644 --- a/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/SpecReader.java +++ b/flexmark-test-util/src/main/java/com/vladsch/flexmark/test/util/spec/SpecReader.java @@ -227,7 +227,7 @@ protected void processLine(String line) { if (!lineAbsorbed) { if (lineProcessed) { comment = null; - } else if (section != null) { + } else if (section != null && state == State.BEFORE) { if (comment == null) comment = new StringBuilder(); comment.append(line).append('\n'); } diff --git a/flexmark-test-util/src/test/java/com/vladsch/flexmark/test/util/TestUtilsTestSuite.java b/flexmark-test-util/src/test/java/com/vladsch/flexmark/test/util/TestUtilsTestSuite.java index 72fa370dc..2b9459290 100644 --- a/flexmark-test-util/src/test/java/com/vladsch/flexmark/test/util/TestUtilsTestSuite.java +++ b/flexmark-test-util/src/test/java/com/vladsch/flexmark/test/util/TestUtilsTestSuite.java @@ -6,6 +6,7 @@ @RunWith(Suite.class) @Suite.SuiteClasses({ LoadUnloadDataKeyAggregatorTest.class, + TestUtilsTest.class }) public class TestUtilsTestSuite { } diff --git a/flexmark-tree-iteration/README.md b/flexmark-tree-iteration/README.md index 363f5749a..49410b723 100644 --- a/flexmark-tree-iteration/README.md +++ b/flexmark-tree-iteration/README.md @@ -4,7 +4,6 @@ flexmark-tree-iteration **tree-iteration-util** Used to iterate, including recursively, over tree structure elements using node class or value filters. - ### Requirements * Java 8 or above @@ -13,9 +12,7 @@ using node class or value filters. * `org.slf4j:slf4j-api:1.7.25` library. * `org.jetbrains:annotations:15.0` library. -### master [![Build status](https://travis-ci.org/vsch/flexmark-java.svg?branch=master)](https://travis-ci.org/vsch/flexmark-java) - -### latest [![Maven Central status](https://img.shields.io/maven-central/v/com.vladsch.flexmark/flexmark.svg)](https://search.maven.org/search?q=g:com.vladsch.flexmark) [![Build status](https://travis-ci.org/vsch/flexmark-java.svg?branch=0.50.40)](https://travis-ci.org/vsch/flexmark-java) [![Javadocs](https://www.javadoc.io/badge/com.vladsch.flexmark/flexmark.svg)](https://www.javadoc.io/doc/com.vladsch.flexmark/flexmark) +### latest [![Maven Central status](https://img.shields.io/maven-central/v/com.vladsch.flexmark/flexmark.svg)](https://search.maven.org/search?q=g:com.vladsch.flexmark) [![Javadocs](https://www.javadoc.io/badge/com.vladsch.flexmark/flexmark.svg)](https://www.javadoc.io/doc/com.vladsch.flexmark/flexmark) ### Examples: @@ -60,16 +57,5 @@ Copyright (c) 2019, Vladimir Schneider , BSD (2-clause) licensed, see [LICENSE.txt] file. [LICENSE.txt]: ../LICENSE.txt -[All about me]: https://vladsch.com/about -[Android Studio]: https://developer.android.com/sdk/installing/studio.html -[GitHub Issues page]: https://github.com/vsch/tree-iteration/issues -[LICENSE]: LICENSE.txt -[Markdown Navigator]: http://vladsch.com/product/markdown-navigator -[Maven Central status]: https://img.shields.io/maven-central/v/com.vladsch.tree-iteration/tree-iteration.svg -[Mia: Missing In Actions]: https://plugins.jetbrains.com/idea/plugin/9257-missing-in-actions -[MultiMarkdown]: https://fletcherpenney.net/multimarkdown/ -[Semantic Versioning]: https://semver.org/ -[tree-iteration]: https://github.com/vsch/tree-iteration -[tree-iteration on Maven]: https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.vladsch.tree-iteration%22 -[tree-iteration wiki]: https://github.com/vsch/tree-iteration/wiki +[tree-iteration on Maven]: https://central.sonatype.com/search?q=g%3Acom.vladsch.tree-iteration&smo=true diff --git a/flexmark-tree-iteration/flexmark-tree-iteration.iml b/flexmark-tree-iteration/flexmark-tree-iteration.iml index 1d273510a..7371edfa1 100644 --- a/flexmark-tree-iteration/flexmark-tree-iteration.iml +++ b/flexmark-tree-iteration/flexmark-tree-iteration.iml @@ -23,6 +23,7 @@ + @@ -30,9 +31,9 @@ - + - + \ No newline at end of file diff --git a/flexmark-tree-iteration/pom.xml b/flexmark-tree-iteration/pom.xml index b7cf8a63a..a0f87c20e 100644 --- a/flexmark-tree-iteration/pom.xml +++ b/flexmark-tree-iteration/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-tree-iteration @@ -15,7 +15,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 com.vladsch.flexmark diff --git a/flexmark-util-ast/flexmark-util-ast.iml b/flexmark-util-ast/flexmark-util-ast.iml index e3c554834..514001d96 100644 --- a/flexmark-util-ast/flexmark-util-ast.iml +++ b/flexmark-util-ast/flexmark-util-ast.iml @@ -25,7 +25,7 @@ - + @@ -34,4 +34,4 @@ - + \ No newline at end of file diff --git a/flexmark-util-ast/pom.xml b/flexmark-util-ast/pom.xml index a9cb598a7..0103aa170 100644 --- a/flexmark-util-ast/pom.xml +++ b/flexmark-util-ast/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-ast @@ -45,7 +45,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/DelimitedNode.java b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/DelimitedNode.java index 374c0f065..9cc4bad4c 100644 --- a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/DelimitedNode.java +++ b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/DelimitedNode.java @@ -21,7 +21,7 @@ public interface DelimitedNode extends TextContainer { void setClosingMarker(BasedSequence closingMarker); @Override - default boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + default boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { if (any(flags, F_NODE_TEXT)) { out.append(getText()); return false; diff --git a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/SpaceInsertingSequenceBuilder.java b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/SpaceInsertingSequenceBuilder.java index bc38e3f28..8eadf6f9e 100644 --- a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/SpaceInsertingSequenceBuilder.java +++ b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/SpaceInsertingSequenceBuilder.java @@ -1,5 +1,6 @@ package com.vladsch.flexmark.util.ast; +import com.vladsch.flexmark.util.misc.BitFieldSet; import com.vladsch.flexmark.util.misc.CharPredicate; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Range; @@ -13,36 +14,38 @@ public class SpaceInsertingSequenceBuilder implements ISequenceBuilder { @NotNull public static SpaceInsertingSequenceBuilder emptyBuilder(@NotNull BasedSequence base) { - return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base)); + return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base), false); } @NotNull public static SpaceInsertingSequenceBuilder emptyBuilder(@NotNull BasedSequence base, @NotNull SegmentOptimizer optimizer) { - return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, optimizer)); + return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, optimizer), false); } @NotNull public static SpaceInsertingSequenceBuilder emptyBuilder(@NotNull BasedSequence base, int options) { - return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, options)); + return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, options), BitFieldSet.any(options, TextContainer.F_ADD_SPACES_BETWEEN_NODES)); } @NotNull public static SpaceInsertingSequenceBuilder emptyBuilder(@NotNull BasedSequence base, int options, @NotNull SegmentOptimizer optimizer) { - return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, options, optimizer)); + return new SpaceInsertingSequenceBuilder(SequenceBuilder.emptyBuilder(base, options, optimizer), BitFieldSet.any(options, TextContainer.F_ADD_SPACES_BETWEEN_NODES)); } @NotNull public static SpaceInsertingSequenceBuilder emptyBuilder(@NotNull SequenceBuilder builder) { - return new SpaceInsertingSequenceBuilder(builder); + return new SpaceInsertingSequenceBuilder(builder, false); } final SequenceBuilder out; Node lastNode; - boolean addSpaceOnNonBlank; boolean needEol; + final boolean addSpacesBetweenNodes; + boolean addSpaces; - private SpaceInsertingSequenceBuilder(SequenceBuilder out) { + private SpaceInsertingSequenceBuilder(SequenceBuilder out, boolean addSpacesBetweenNodes) { this.out = out; + this.addSpacesBetweenNodes = addSpacesBetweenNodes; } public SequenceBuilder getOut() { @@ -54,14 +57,6 @@ public char charAt(int index) { return out.charAt(index); } - public boolean isAddSpaceOnNonBlank() { - return addSpaceOnNonBlank; - } - - public void setAddSpaceOnNonBlank(boolean addSpaceOnNonBlank) { - this.addSpaceOnNonBlank = addSpaceOnNonBlank; - } - public boolean isNeedEol() { return needEol; } @@ -79,10 +74,10 @@ public void setLastNode(Node lastNode) { if (this.lastNode != null && this.lastNode.getEndOffset() < lastNode.getStartOffset()) { BasedSequence sequence = getBaseSequence().subSequence(this.lastNode.getEndOffset(), lastNode.getStartOffset()); - this.addSpaceOnNonBlank = sequence.indexOfAny(CharPredicate.SPACE_TAB_EOL) != -1; this.needEol = sequence.trim(CharPredicate.SPACE_TAB).length() > 0 && sequence.trim(CharPredicate.WHITESPACE).isEmpty(); } + addSpaces = addSpacesBetweenNodes; this.lastNode = lastNode; } @@ -155,14 +150,14 @@ public boolean needEol() { @Override @NotNull - public SpaceInsertingSequenceBuilder getBuilder() {return new SpaceInsertingSequenceBuilder(out.getBuilder());} + public SpaceInsertingSequenceBuilder getBuilder() {return new SpaceInsertingSequenceBuilder(out.getBuilder(), addSpacesBetweenNodes);} @Override @NotNull public SpaceInsertingSequenceBuilder append(@Nullable CharSequence chars, int startIndex, int endIndex) { - if (addSpaceOnNonBlank && chars != null && startIndex < endIndex && !CharPredicate.WHITESPACE.test(chars.charAt(startIndex)) && needSpace()) { + if (addSpaces && chars != null && startIndex < endIndex && !CharPredicate.WHITESPACE.test(chars.charAt(startIndex)) && needSpace()) { out.append(' '); - addSpaceOnNonBlank = false; + addSpaces = false; } out.append(chars, startIndex, endIndex); return this; @@ -171,9 +166,9 @@ public SpaceInsertingSequenceBuilder append(@Nullable CharSequence chars, int st @Override @NotNull public SpaceInsertingSequenceBuilder append(char c) { - if (addSpaceOnNonBlank && !CharPredicate.WHITESPACE.test(c) && needSpace()) { + if (addSpaces && !CharPredicate.WHITESPACE.test(c) && needSpace()) { out.append(' '); - addSpaceOnNonBlank = false; + addSpaces = false; } out.append(c); return this; @@ -182,9 +177,9 @@ public SpaceInsertingSequenceBuilder append(char c) { @Override @NotNull public SpaceInsertingSequenceBuilder append(char c, int count) { - if (addSpaceOnNonBlank && !CharPredicate.WHITESPACE.test(c) && needSpace()) { + if (addSpaces && !CharPredicate.WHITESPACE.test(c) && needSpace()) { out.append(' '); - addSpaceOnNonBlank = false; + addSpaces = false; } out.append(c, count); return this; @@ -192,9 +187,9 @@ public SpaceInsertingSequenceBuilder append(char c, int count) { @NotNull public SpaceInsertingSequenceBuilder append(int startOffset, int endOffset) { - if (addSpaceOnNonBlank && startOffset < endOffset && !CharPredicate.WHITESPACE.test(out.getBaseSequence().charAt(startOffset)) && needSpace()) { + if (addSpaces && startOffset < endOffset && !CharPredicate.WHITESPACE.test(out.getBaseSequence().charAt(startOffset)) && needSpace()) { out.append(' '); - addSpaceOnNonBlank = false; + addSpaces = false; } out.append(startOffset, endOffset); return this; diff --git a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextCollectingVisitor.java b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextCollectingVisitor.java index 3a990d81d..60e9c8bfc 100644 --- a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextCollectingVisitor.java +++ b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextCollectingVisitor.java @@ -8,33 +8,31 @@ public class TextCollectingVisitor { final private NodeVisitor myVisitor; SpaceInsertingSequenceBuilder out; - int flags; + int flags; // flags defined by TextContainer public TextCollectingVisitor() { myVisitor = new NodeVisitor() { @Override public void processNode(@NotNull Node node, boolean withChildren, @NotNull BiConsumer> processor) { if (!node.isOrDescendantOfType(DoNotCollectText.class)) { + out.setLastNode(node); + if (node instanceof Block && out.isNotEmpty()) { + out.appendEol(); + } if (node instanceof TextContainer) { - out.setLastNode(node); - if (((TextContainer) node).collectText(out, flags)) { + final TextContainer textContainer = (TextContainer) node; + if (textContainer.collectText(out, flags, myVisitor)) { if (node instanceof BlankLineBreakNode && out.isNotEmpty()) { out.appendEol(); } processChildren(node, processor); - if (node instanceof LineBreakNode && out.needEol()) { - out.appendEol(); - } } + textContainer.collectEndText(out, flags, myVisitor); } else { - out.setLastNode(node); - if (node instanceof BlankLineBreakNode && out.isNotEmpty()) { - out.appendEol(); - } processChildren(node, processor); - if (node instanceof LineBreakNode && out.needEol()) { - out.appendEol(); - } + } + if (node instanceof LineBreakNode && out.needEol()) { + out.appendEol(); } } } @@ -62,7 +60,7 @@ public BasedSequence collectAndGetSequence(Node node) { } public void collect(Node node, int flags) { - out = SpaceInsertingSequenceBuilder.emptyBuilder(node.getChars()); + out = SpaceInsertingSequenceBuilder.emptyBuilder(node.getChars(), flags); this.flags = flags; myVisitor.visit(node); } diff --git a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextContainer.java b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextContainer.java index fbf8c5cec..064608a45 100644 --- a/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextContainer.java +++ b/flexmark-util-ast/src/main/java/com/vladsch/flexmark/util/ast/TextContainer.java @@ -9,6 +9,10 @@ public interface TextContainer { enum Flags implements BitField { LINK_TEXT_TYPE(3), NODE_TEXT, // not unescaped and not percent decoded + FOR_HEADING_ID, // text for heading ID + NO_TRIM_REF_TEXT_START, // don't trim ref text start + NO_TRIM_REF_TEXT_END, // don't trim ref text end + ADD_SPACES_BETWEEN_NODES, // when appending text from different nodes, ensure there is at least one space ; final int bits; @@ -28,20 +32,35 @@ public int getBits() { } int F_LINK_TEXT_TYPE = BitFieldSet.intMask(Flags.LINK_TEXT_TYPE); - int F_LINK_TEXT = 0; - int F_LINK_PAGE_REF = 1; - int F_LINK_ANCHOR = 2; - int F_LINK_URL = 3; - int F_LINK_NODE_TEXT = 4; + int F_LINK_TEXT = 0; // use link text + int F_LINK_PAGE_REF = 1; // use page ref + int F_LINK_ANCHOR = 2; // use link anchor + int F_LINK_URL = 3; // use link URL + int F_LINK_NODE_TEXT = 4; // use node text int F_NODE_TEXT = BitFieldSet.intMask(Flags.NODE_TEXT); + int F_FOR_HEADING_ID = BitFieldSet.intMask(Flags.FOR_HEADING_ID); + int F_NO_TRIM_REF_TEXT_START = BitFieldSet.intMask(Flags.NO_TRIM_REF_TEXT_START); + int F_NO_TRIM_REF_TEXT_END = BitFieldSet.intMask(Flags.NO_TRIM_REF_TEXT_END); + int F_ADD_SPACES_BETWEEN_NODES = BitFieldSet.intMask(Flags.ADD_SPACES_BETWEEN_NODES); /** * Append node's text * * @param out sequence build to which to append text * @param flags collection flags + * @param nodeVisitor node visitor to use to visit children * @return true if child nodes should be visited */ - boolean collectText(ISequenceBuilder, BasedSequence> out, int flags); + boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor); + + /** + * Append node's text ending, after any child nodes have been visited. + * The default implementation does nothing. + * + * @param out sequence build to which to append text + * @param flags collection flags + * @param nodeVisitor node visitor to use to visit children + */ + default void collectEndText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) {} } diff --git a/flexmark-util-builder/flexmark-util-builder.iml b/flexmark-util-builder/flexmark-util-builder.iml index 4e83d926c..a0c8cc3da 100644 --- a/flexmark-util-builder/flexmark-util-builder.iml +++ b/flexmark-util-builder/flexmark-util-builder.iml @@ -25,7 +25,7 @@ - + diff --git a/flexmark-util-builder/pom.xml b/flexmark-util-builder/pom.xml index ce9c1767c..bdbd92586 100644 --- a/flexmark-util-builder/pom.xml +++ b/flexmark-util-builder/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-builder @@ -29,7 +29,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-builder/src/main/java/com/vladsch/flexmark/util/builder/BuilderBase.java b/flexmark-util-builder/src/main/java/com/vladsch/flexmark/util/builder/BuilderBase.java index ddd07069c..74c28eac3 100644 --- a/flexmark-util-builder/src/main/java/com/vladsch/flexmark/util/builder/BuilderBase.java +++ b/flexmark-util-builder/src/main/java/com/vladsch/flexmark/util/builder/BuilderBase.java @@ -138,7 +138,7 @@ public V get(@NotNull DataKey key) { return key.get(this); } - protected BuilderBase(DataHolder options) { + protected BuilderBase(@Nullable DataHolder options) { super(options); } diff --git a/flexmark-util-collection/flexmark-util-collection.iml b/flexmark-util-collection/flexmark-util-collection.iml index 115f9b8ad..2a9525dc9 100644 --- a/flexmark-util-collection/flexmark-util-collection.iml +++ b/flexmark-util-collection/flexmark-util-collection.iml @@ -25,9 +25,9 @@ - + - - + + \ No newline at end of file diff --git a/flexmark-util-collection/pom.xml b/flexmark-util-collection/pom.xml index 69e602535..77c980b4f 100644 --- a/flexmark-util-collection/pom.xml +++ b/flexmark-util-collection/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-collection @@ -24,7 +24,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-data/flexmark-util-data.iml b/flexmark-util-data/flexmark-util-data.iml index a80e91449..69ea2044f 100644 --- a/flexmark-util-data/flexmark-util-data.iml +++ b/flexmark-util-data/flexmark-util-data.iml @@ -25,9 +25,9 @@ - + - - + + \ No newline at end of file diff --git a/flexmark-util-data/pom.xml b/flexmark-util-data/pom.xml index f5d532500..ea0491bfc 100644 --- a/flexmark-util-data/pom.xml +++ b/flexmark-util-data/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-data @@ -25,7 +25,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/MutableDataSet.java b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/MutableDataSet.java index 36b31b6be..4d16a604c 100644 --- a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/MutableDataSet.java +++ b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/MutableDataSet.java @@ -8,7 +8,7 @@ public MutableDataSet() { super(); } - public MutableDataSet(DataHolder other) { + public MutableDataSet(@Nullable DataHolder other) { super(other); } diff --git a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/ScopedDataSet.java b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/ScopedDataSet.java index 9aca45114..6e55c2621 100644 --- a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/ScopedDataSet.java +++ b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/ScopedDataSet.java @@ -3,20 +3,17 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; -import java.util.ArrayList; -import java.util.Collection; -import java.util.HashMap; -import java.util.Map; +import java.util.*; public class ScopedDataSet extends DataSet { protected final DataHolder parent; - public ScopedDataSet(DataHolder parent) { + public ScopedDataSet(@Nullable DataHolder parent) { super(); this.parent = parent; } - public ScopedDataSet(DataHolder parent, DataHolder other) { + public ScopedDataSet(@Nullable DataHolder parent, @Nullable DataHolder other) { super(other); this.parent = parent; } @@ -28,15 +25,8 @@ public DataHolder getParent() { @Override public @NotNull Map, Object> getAll() { if (parent != null) { - - HashMap, Object> all = new HashMap<>(super.getAll()); - - for (DataKeyBase key : parent.getKeys()) { - if (!contains(key)) { - all.put(key, key.get(parent)); - } - } - + HashMap, Object> all = new HashMap<>(parent.getAll()); + all.putAll(super.getAll()); return all; } else { return super.getAll(); @@ -46,21 +36,21 @@ public DataHolder getParent() { @Override public @NotNull Collection> getKeys() { if (parent != null) { - - ArrayList> all = new ArrayList<>(super.getKeys()); - - for (DataKeyBase key : parent.getKeys()) { - if (!contains(key)) { - all.add(key); - } - } - + HashSet> all = new HashSet<>(parent.getKeys()); + all.addAll(super.getKeys()); return all; } else { return super.getKeys(); } } + @Override + public @NotNull MutableDataSet toMutable() { + MutableDataSet mutableDataSet = new MutableDataSet(); + mutableDataSet.dataSet.putAll(super.getAll()); + return parent != null ? new MutableScopedDataSet(parent, mutableDataSet) : mutableDataSet; + } + @Override public boolean contains(@NotNull DataKeyBase key) { return super.contains(key) || (parent != null && parent.contains(key)); diff --git a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/SharedDataKeys.java b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/SharedDataKeys.java index cbd8e5c5f..004f884ad 100644 --- a/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/SharedDataKeys.java +++ b/flexmark-util-data/src/main/java/com/vladsch/flexmark/util/data/SharedDataKeys.java @@ -30,6 +30,9 @@ public class SharedDataKeys { final public static DataKey HEADER_ID_GENERATOR_NON_DASH_CHARS = new DataKey<>("HEADER_ID_GENERATOR_NON_DASH_CHARS", ""); final public static DataKey HEADER_ID_GENERATOR_NO_DUPED_DASHES = new DataKey<>("HEADER_ID_GENERATOR_NO_DUPED_DASHES", false); final public static DataKey HEADER_ID_GENERATOR_NON_ASCII_TO_LOWERCASE = new DataKey<>("HEADER_ID_GENERATOR_NON_ASCII_TO_LOWERCASE", true); + final public static DataKey HEADER_ID_REF_TEXT_TRIM_LEADING_SPACES = new DataKey<>("HEADER_ID_REF_TEXT_TRIM_LEADING_SPACES", true); + final public static DataKey HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES = new DataKey<>("HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES", true); + final public static DataKey HEADER_ID_ADD_EMOJI_SHORTCUT = new DataKey<>("HEADER_ID_ADD_EMOJI_SHORTCUT", false); final public static DataKey RENDER_HEADER_ID = new DataKey<>("RENDER_HEADER_ID", false); final public static DataKey GENERATE_HEADER_ID = new DataKey<>("GENERATE_HEADER_ID", true); final public static DataKey DO_NOT_RENDER_LINKS = new DataKey<>("DO_NOT_RENDER_LINKS", false); diff --git a/flexmark-util-dependency/flexmark-util-dependency.iml b/flexmark-util-dependency/flexmark-util-dependency.iml index eb02ffd9b..5100f96fc 100644 --- a/flexmark-util-dependency/flexmark-util-dependency.iml +++ b/flexmark-util-dependency/flexmark-util-dependency.iml @@ -25,11 +25,11 @@ - + - + \ No newline at end of file diff --git a/flexmark-util-dependency/pom.xml b/flexmark-util-dependency/pom.xml index 22638757b..b20ab1272 100644 --- a/flexmark-util-dependency/pom.xml +++ b/flexmark-util-dependency/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-dependency @@ -33,7 +33,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-dependency/src/main/java/com/vladsch/flexmark/util/dependency/FlatDependencyHandler.java b/flexmark-util-dependency/src/main/java/com/vladsch/flexmark/util/dependency/FlatDependencyHandler.java index 0e184bd85..e883de085 100644 --- a/flexmark-util-dependency/src/main/java/com/vladsch/flexmark/util/dependency/FlatDependencyHandler.java +++ b/flexmark-util-dependency/src/main/java/com/vladsch/flexmark/util/dependency/FlatDependencyHandler.java @@ -6,7 +6,7 @@ import java.util.function.Function; /** - * @deprecated use {@link DependencyResolver#resolveFlatDependencies(List, Function, Function)} + * @deprecated use {@link DependencyResolver#resolveFlatDependencies(List, Function, Function)} use null for functions if no need for sort or class extractor */ @Deprecated public class FlatDependencyHandler extends DependencyHandler, FlatDependencies> { diff --git a/flexmark-util-experimental/flexmark-util-experimental.iml b/flexmark-util-experimental/flexmark-util-experimental.iml index 7721ed921..88cef5162 100644 --- a/flexmark-util-experimental/flexmark-util-experimental.iml +++ b/flexmark-util-experimental/flexmark-util-experimental.iml @@ -29,7 +29,7 @@ - + @@ -39,4 +39,4 @@ - + \ No newline at end of file diff --git a/flexmark-util-experimental/pom.xml b/flexmark-util-experimental/pom.xml index 66c615383..9e19243c4 100644 --- a/flexmark-util-experimental/pom.xml +++ b/flexmark-util-experimental/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-experimental @@ -45,7 +45,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-format/flexmark-util-format.iml b/flexmark-util-format/flexmark-util-format.iml index aa52e7387..307d4b1aa 100644 --- a/flexmark-util-format/flexmark-util-format.iml +++ b/flexmark-util-format/flexmark-util-format.iml @@ -25,7 +25,7 @@ - + diff --git a/flexmark-util-format/pom.xml b/flexmark-util-format/pom.xml index 556eb973e..6ecd5f74d 100644 --- a/flexmark-util-format/pom.xml +++ b/flexmark-util-format/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-format @@ -50,7 +50,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownParagraph.java b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownParagraph.java index e5cd8aaef..ccf930b9b 100644 --- a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownParagraph.java +++ b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownParagraph.java @@ -10,6 +10,7 @@ import com.vladsch.flexmark.util.sequence.builder.SequenceBuilder; import com.vladsch.flexmark.util.sequence.builder.tree.BasedOffsetTracker; import com.vladsch.flexmark.util.sequence.builder.tree.OffsetInfo; +import com.vladsch.flexmark.util.sequence.mappers.SpaceMapper; import com.vladsch.flexmark.util.sequence.mappers.SpecialLeadInHandler; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; @@ -77,13 +78,13 @@ public Range getContinuationStartSplice(int offset, boolean afterSpace, boolean if (afterSpace && afterDelete) { BasedOffsetTracker preFormatTracker = BasedOffsetTracker.create(altSeq); int startOfLine = baseSequence.startOfLine(offset); - if (startOfLine > altSeq.getStartOffset() && !baseSequence.isCharAt(offset, CharPredicate.SPACE_TAB_LINE_SEP)) { - int previousNonBlank = baseSequence.lastIndexOfAnyNot(CharPredicate.SPACE_TAB_EOL, offset - 1); + if (startOfLine > altSeq.getStartOffset() && !baseSequence.isCharAt(offset, SPACE_TAB_NBSP_LINE_SEP)) { + int previousNonBlank = baseSequence.lastIndexOfAnyNot(SPACE_TAB_NBSP_EOL, offset - 1); if (previousNonBlank < startOfLine) { // delete range between last non-blank and offset index @NotNull OffsetInfo offsetInfo = preFormatTracker.getOffsetInfo(offset, true); int offsetIndex = offsetInfo.endIndex; - int previousNonBlankIndex = altSeq.lastIndexOfAnyNot(CharPredicate.SPACE_TAB_EOL, offsetIndex - 1); + int previousNonBlankIndex = altSeq.lastIndexOfAnyNot(SPACE_TAB_NBSP_EOL, offsetIndex - 1); return Range.of(previousNonBlankIndex + 1, offsetIndex); } } @@ -99,13 +100,13 @@ BasedSequence resolveTrackedOffsets(@NotNull BasedSequence unwrapped, @NotNull B for (int i = iMax; i-- > 0; ) { TrackedOffset trackedOffset = trackedOffsets.get(i); int offset = trackedOffset.getOffset(); - boolean baseIsWhiteSpaceAtOffset = unwrapped.isBaseCharAt(offset, WHITESPACE); + boolean baseIsWhiteSpaceAtOffset = unwrapped.isBaseCharAt(offset, WHITESPACE_NBSP); - if (baseIsWhiteSpaceAtOffset && !(unwrapped.isBaseCharAt(offset - 1, WHITESPACE))) { + if (baseIsWhiteSpaceAtOffset && !(unwrapped.isBaseCharAt(offset - 1, WHITESPACE_NBSP))) { // we need to use previous non-blank and use that offset OffsetInfo info = tracker.getOffsetInfo(offset - 1, false); trackedOffset.setIndex(info.endIndex); - } else if (!baseIsWhiteSpaceAtOffset && unwrapped.isBaseCharAt(offset + 1, WHITESPACE)) { + } else if (!baseIsWhiteSpaceAtOffset && unwrapped.isBaseCharAt(offset + 1, WHITESPACE_NBSP)) { // we need to use this non-blank and use that offset OffsetInfo info = tracker.getOffsetInfo(offset, false); trackedOffset.setIndex(info.startIndex); @@ -182,8 +183,10 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence TrackedOffset trackedOffset = trackedOffsets.get(i); int offset = trackedOffset.getOffset(); - int countedSpacesBefore = baseSequence.countTrailingSpaceTab(offset); - int countedSpacesAfter = baseSequence.countLeadingSpaceTab(offset); + int countedSpacesBefore = baseSequence.countTrailing(SPACE_TAB_NBSP, offset); + int countedSpacesAfter = baseSequence.countLeading(SPACE_TAB_NBSP, offset); + int countedWhitespaceBefore = baseSequence.countTrailing(SPACE_TAB_NBSP_EOL, offset); + int countedWhiteSpaceAfter = baseSequence.countLeading(SPACE_TAB_NBSP_EOL, offset); if (inTest) { assert trackedOffset.getSpacesBefore() == countedSpacesBefore; @@ -204,7 +207,7 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence System.out.println(trackedOffset); } - if (!CharPredicate.SPACE_TAB.test(baseCharAt)) { + if (!SPACE_TAB_NBSP.test(baseCharAt)) { anchorOffset = offset; anchorIndex = tracker.getOffsetInfo(anchorOffset, false).startIndex; @@ -214,12 +217,12 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence anchorResolvedBy = "LSep "; } } else { - if (!CharPredicate.SPACE_TAB_EOL.test(prevBaseCharAt)) { - anchorOffset = offset - countedSpacesBefore; + if (!SPACE_TAB_NBSP_EOL.test(prevBaseCharAt)) { + anchorOffset = offset - countedWhitespaceBefore; anchorIndex = tracker.getOffsetInfo(anchorOffset - 1, false).endIndex; anchorResolvedBy = "Prev "; - } else if (!CharPredicate.SPACE_TAB_EOL.test(nextBaseCharAt)) { - anchorOffset = offset + countedSpacesAfter; + } else if (!SPACE_TAB_NBSP_EOL.test(nextBaseCharAt)) { + anchorOffset = offset + countedWhiteSpaceAfter; anchorIndex = tracker.getOffsetInfo(anchorOffset, false).startIndex; anchorResolvedBy = "Next "; } else { @@ -233,14 +236,6 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence System.out.println(String.format("%saltUnwrapped anchor: `%s`", anchorResolvedBy, altUnwrapped.safeSubSequence(anchorIndex - 10, anchorIndex).toVisibleWhitespaceString() + "|" + altUnwrapped.safeSubSequence(anchorIndex, anchorIndex + 10).toVisibleWhitespaceString())); } - assert baseSequence.safeCharAt(anchorOffset) == altUnwrapped.safeCharAt(anchorIndex + anchorDelta) - // NOTE: alt sequence could have spaces removed which would result in space in base sequence and EOL in alt sequence - || baseSequence.isCharAt(anchorOffset, WHITESPACE) && altUnwrapped.isCharAt(anchorIndex + anchorDelta, WHITESPACE_OR_NUL) - : String.format("baseSeq.charAt: %d != altUnwrapped.charAt: %d" - , (int) baseSequence.safeCharAt(anchorOffset) - , (int) altUnwrapped.safeCharAt(anchorIndex + anchorDelta) - ); - // now see where it is in the wrapped sequence int wrappedIndex = altTracker.getOffsetInfo(anchorOffset, false).startIndex; @@ -249,18 +244,52 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence System.out.println(String.format("wrapped anchor: `%s`", wrapped.safeSubSequence(wrappedIndex - 10, wrappedIndex).toVisibleWhitespaceString() + "|" + wrapped.safeSubSequence(wrappedIndex, wrappedIndex + 10).toVisibleWhitespaceString())); } + // NOTE: at this point space == since altUnwrapped can have instead of spaces + assert (SpaceMapper.areEquivalent(baseSequence.safeCharAt(anchorOffset), altUnwrapped.safeCharAt(anchorIndex + anchorDelta)) + // NOTE: alt sequence could have spaces removed which would result in space in base sequence and EOL in alt sequence, also unwrapped can have spaces replaced with NBSP + || baseSequence.isCharAt(anchorOffset, WHITESPACE_NBSP_OR_NUL) && altUnwrapped.isCharAt(anchorIndex + anchorDelta, WHITESPACE_NBSP_OR_NUL)) + : String.format("baseSeq.charAt(%d): '%s':0x%04x != altUnwrapped.charAt(%d=%d+%d): '%s':0x%04x, baseSequence anchor: '%s', altUnwrapped anchor: '%s', altWrapped anchor: '%s', wrapped anchor: '%s'" + , anchorOffset + , String.valueOf(baseSequence.safeCharAt(anchorOffset)) + , (int) baseSequence.safeCharAt(anchorOffset) + , anchorIndex + anchorDelta + , anchorIndex + , anchorDelta + , String.valueOf(altUnwrapped.safeCharAt(anchorIndex + anchorDelta)) + , (int) altUnwrapped.safeCharAt(anchorIndex + anchorDelta) + , baseSequence.safeSubSequence(anchorOffset - 10, anchorOffset).toVisibleWhitespaceString() + "|" + baseSequence.safeSubSequence(anchorOffset, anchorOffset + 10).toVisibleWhitespaceString() + , altUnwrapped.safeSubSequence(anchorIndex + anchorDelta - 10, anchorIndex + anchorDelta).toVisibleWhitespaceString() + "|" + altUnwrapped.safeSubSequence(anchorIndex + anchorDelta, anchorIndex + anchorDelta + 10).toVisibleWhitespaceString() + , altWrapped.safeSubSequence(wrappedIndex - 10, wrappedIndex).toVisibleWhitespaceString() + "|" + altWrapped.safeSubSequence(wrappedIndex, wrappedIndex + 10).toVisibleWhitespaceString() + , wrapped.safeSubSequence(wrappedIndex - 10, wrappedIndex).toVisibleWhitespaceString() + "|" + wrapped.safeSubSequence(wrappedIndex, wrappedIndex + 10).toVisibleWhitespaceString() + ); + int wrappedAdjusted = 0; + int unwrappedAdjusted = 0; + int addSpacesBeforeEol = 0; // take char start but if at whitespace, use previous for validation - if (WHITESPACE.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta))) { - wrappedAdjusted = -1; + if (WHITESPACE_NBSP.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta))) { + // NOTE: if after wrapping caret is still on whitespace or end of string, then we adjust unwrapped and wrapped index backwards + if (WHITESPACE_NBSP_OR_NUL.test(altWrapped.safeCharAt(wrappedIndex))) { + unwrappedAdjusted = -1; + wrappedAdjusted = -1; + } else { + // NOTE: if the insert pos is on EOL (or whitespace) but after wrapping the leading whitespace is removed + // then need to adjust only unwrapped and add space after tracked pos: + // altUnwrapped anchor: `en updated|\nabandoned` + // altWrapped anchor: `n\nupdated |abandoned ` + // wrapped anchor: `n\nupdated |abandoned ` + addSpacesBeforeEol = 1; + unwrappedAdjusted = altUnwrapped.countLeading(WHITESPACE_NBSP, anchorIndex + anchorDelta); + } } else if (altUnwrapped.safeCharAt(anchorIndex + anchorDelta) == SequenceUtils.LS) { // have line sep at anchor, if prev is not whitespace, use it for validation - if (!WHITESPACE.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta - 1))) { + if (!WHITESPACE_NBSP.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta - 1))) { wrappedAdjusted--; + unwrappedAdjusted--; } else { // use next char, it should not be whitespace anchorDelta++; - assert !WHITESPACE.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta)) : + assert !WHITESPACE_NBSP.test(altUnwrapped.safeCharAt(anchorIndex + anchorDelta)) : String.format("Character(%s) after LS should not be whitespace.", SequenceUtils.toVisibleWhitespaceString(Character.toString(altUnwrapped.safeCharAt(anchorIndex + anchorDelta))) ); @@ -268,14 +297,27 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence } } - char altUnwrappedCharAt = altUnwrapped.safeCharAt(anchorIndex + anchorDelta + wrappedAdjusted); + if (inTest) { + int useIndex = anchorIndex + anchorDelta + unwrappedAdjusted; + System.out.println(String.format("adjusted altWrapped anchor: `%s`", altWrapped.safeSubSequence(useIndex - 10, useIndex).toVisibleWhitespaceString() + "|" + altWrapped.safeSubSequence(useIndex, useIndex + 10).toVisibleWhitespaceString())); + int useWrapIndex = wrappedIndex + wrappedAdjusted; + System.out.println(String.format("adjusted wrapped anchor: `%s`", wrapped.safeSubSequence(useWrapIndex - 10, useWrapIndex).toVisibleWhitespaceString() + "|" + wrapped.safeSubSequence(useWrapIndex, useWrapIndex + 10).toVisibleWhitespaceString())); + } + + char altUnwrappedCharAt = altUnwrapped.safeCharAt(anchorIndex + anchorDelta + unwrappedAdjusted); char wrappedCharAt = wrapped.safeCharAt(wrappedIndex + wrappedAdjusted); - assert altUnwrappedCharAt == wrappedCharAt - || WHITESPACE.test(altUnwrappedCharAt) && WHITESPACE.test(wrappedCharAt) - : String.format("altUnwrapped.charAt: %s != wrapped.charAt: %s" + // NOTE: at this point space == since altUnwrapped can have instead of spaces + assert SpaceMapper.areEquivalent(altUnwrappedCharAt, wrappedCharAt) + || WHITESPACE_NBSP.test(altUnwrappedCharAt) && WHITESPACE_NBSP.test(wrappedCharAt) + : String.format("altUnwrapped.charAt: '%s'(%d) != wrapped.charAt: '%s'(%d) for width=%d, unwrapped: '%s', wrapped: '%s'" , SequenceUtils.toVisibleWhitespaceString(Character.toString(altUnwrappedCharAt)) + , (int) altUnwrappedCharAt , SequenceUtils.toVisibleWhitespaceString(Character.toString(wrappedCharAt)) + , (int) wrappedCharAt + , width + , SequenceUtils.toVisibleWhitespaceString(altUnwrapped) + , SequenceUtils.toVisibleWhitespaceString(altWrapped) ); // Adjust index position and restore spaces if needed @@ -291,8 +333,9 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence wrappedIndex -= wrapped.eolEndLength(wrappedIndex); } - int wrappedSpacesBefore = wrapped.countTrailingSpaceTab(wrappedIndex); - int wrappedSpacesAfter = wrapped.countLeadingSpaceTab(wrappedIndex); + // treat NBSP as spaces + int wrappedSpacesBefore = wrapped.countTrailing(SPACE_TAB_NBSP, wrappedIndex); + int wrappedSpacesAfter = wrapped.countLeading(SPACE_TAB_NBSP, wrappedIndex); if (trackedOffset.isAfterSpaceEdit()) { if (trackedOffset.isAfterInsert()) { @@ -304,13 +347,14 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence } int addSpacesBefore = trackedOffset.isSpliced() ? 0 : Math.max(0, countedSpacesBefore - wrappedSpacesBefore); - int addSpacesAfter = Math.max(0, countedSpacesAfter - wrappedSpacesAfter); + // add an implicit space if was followed by EOL + int addSpacesAfter = Math.max(addSpacesBeforeEol, countedSpacesAfter - wrappedSpacesAfter); - if (wrapped.isCharAt(wrappedIndex, ANY_EOL_NUL)) { + if (wrapped.isCharAt(wrappedIndex, CharPredicate.ANY_EOL_NUL)) { // at end of line add only before, nothing after addSpacesAfter = 0; if (trackedOffset.isAfterDelete()) addSpacesBefore = Math.min(1, addSpacesBefore); - } else if (!wrapped.isCharAt(wrappedIndex - 1, ANY_EOL_NUL)) { + } else if (!wrapped.isCharAt(wrappedIndex - 1, CharPredicate.ANY_EOL_NUL)) { // not at start of line // spaces before caret, see if need to add max 1, and all spaces after addSpacesBefore = Math.min(1, addSpacesBefore); @@ -326,8 +370,8 @@ BasedSequence resolveTrackedOffsetsEdit(BasedSequence baseSpliced, BasedSequence } if (addSpacesBefore + addSpacesAfter > 0) { - int lastNonBlank = wrapped.lastIndexOfAnyNot(WHITESPACE); - if (wrappedIndex < lastNonBlank) { + int lastNonBlank = wrapped.lastIndexOfAnyNot(WHITESPACE_NBSP); + if (wrappedIndex <= lastNonBlank) { // insert in middle wrapped = wrapped.insert(wrappedIndex, RepeatedSequence.ofSpaces(addSpacesBefore + addSpacesAfter)); diff --git a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownTable.java b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownTable.java index a4cdb326e..abb83be2d 100644 --- a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownTable.java +++ b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/MarkdownTable.java @@ -992,8 +992,10 @@ public MarkdownTable sorted(ColumnSort[] columnSorts, int textCollectionFlags, @ for (ColumnSort columnSort : columnSorts) { int c = columnSort.column; if (c >= 0 && c < cMax) { - IndexSpanOffset spanIndex = rows.get(r).indexOf(c); - TableCell cell = rows.get(r).cells.get(spanIndex.index); + IndexSpanOffset spanIndex = rows.get(r) + .indexOf(c); + TableCell cell = rows.get(r).cells + .get(spanIndex.index); if (spanIndex.index == c && cell != null) { // not in span cellSizes[c] = Math.max(cellSizes[c], cell.text.length()); diff --git a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/NodeContext.java b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/NodeContext.java index 0d3a423fa..cec8e155d 100644 --- a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/NodeContext.java +++ b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/NodeContext.java @@ -12,7 +12,7 @@ public interface NodeContext> { * * @return a new rendering context with a given appendable for its output */ - C getSubContext(); + @NotNull C getSubContext(); /** * Creates a child rendering context that can be used to collect rendered html text. The child context inherits @@ -21,7 +21,7 @@ public interface NodeContext> { * @param options options to use for the context (only options which do not affect the context construction will be used) * @return a new rendering context with a given appendable for its output */ - C getSubContext(DataHolder options); + @NotNull C getSubContext(@Nullable DataHolder options); /** * Creates a child rendering context that can be used to collect rendered html text. The child context inherits @@ -31,17 +31,17 @@ public interface NodeContext> { * @param builder sequence builder to user for appended text for tracking original base offsets * @return a new rendering context with a given appendable for its output */ - C getSubContext(DataHolder options, @NotNull ISequenceBuilder builder); + @NotNull C getSubContext(@Nullable DataHolder options, @NotNull ISequenceBuilder builder); /** * @return the current node being rendered */ - N getCurrentNode(); + @Nullable N getCurrentNode(); /** * Get options for the context * * @return data holder */ - @Nullable DataHolder getOptions(); + @NotNull DataHolder getOptions(); } diff --git a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/TableFormatOptions.java b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/TableFormatOptions.java index 45df961c6..0c1aab614 100644 --- a/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/TableFormatOptions.java +++ b/flexmark-util-format/src/main/java/com/vladsch/flexmark/util/format/TableFormatOptions.java @@ -9,6 +9,7 @@ @SuppressWarnings("WeakerAccess") public class TableFormatOptions implements MutableDataSetter { + // NOTE: the value of \u001f is hardcoded in Parsing patterns final public static char INTELLIJ_DUMMY_IDENTIFIER_CHAR = SequenceUtils.US; final public static String INTELLIJ_DUMMY_IDENTIFIER = SequenceUtils.US_CHARS; final public static CharPredicate INTELLIJ_DUMMY_IDENTIFIER_SET = value -> value == SequenceUtils.US; diff --git a/flexmark-util-html/flexmark-util-html.iml b/flexmark-util-html/flexmark-util-html.iml index b1c6784e6..bc1a642be 100644 --- a/flexmark-util-html/flexmark-util-html.iml +++ b/flexmark-util-html/flexmark-util-html.iml @@ -26,7 +26,7 @@ - + diff --git a/flexmark-util-html/pom.xml b/flexmark-util-html/pom.xml index 73f8f54ec..01d94299c 100644 --- a/flexmark-util-html/pom.xml +++ b/flexmark-util-html/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-html @@ -28,7 +28,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/Attributes.java b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/Attributes.java index b5c0adf4e..8c2dbe65c 100644 --- a/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/Attributes.java +++ b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/Attributes.java @@ -6,7 +6,7 @@ public class Attributes { final public static Attributes EMPTY = new Attributes(); - private LinkedHashMap attributes; + protected LinkedHashMap attributes; public Attributes() { attributes = null; @@ -15,12 +15,13 @@ public Attributes() { public Attributes(Attributes attributes) { this.attributes = attributes == null || attributes.attributes == null ? null : new LinkedHashMap<>(attributes.attributes); } + + public MutableAttributes toMutable() { + return new MutableAttributes(this); + } - protected LinkedHashMap getAttributes() { - if (attributes == null) { - attributes = new LinkedHashMap<>(); - } - return attributes; + public Attributes toImmutable() { + return this; } public Attribute get(CharSequence key) { @@ -39,76 +40,6 @@ public String getValue(CharSequence key) { return attribute.getValue(); } - public Attribute replaceValue(Attribute attribute) { - return replaceValue(attribute.getName(), attribute.getValue()); - } - - /** - * Attribute dependent value replacement - * class and style append new values to existing ones - * others set it to the new value - * - * @param key attribute name - * @param value new value - * @return new attribute - */ - public Attribute replaceValue(CharSequence key, CharSequence value) { - String useKey = String.valueOf(key); - Attribute attribute; - if (attributes == null) { - attribute = AttributeImpl.of(useKey, value); - } else { - attribute = attributes.get(useKey); - if (attribute != null) attribute = attribute.replaceValue(value); - else attribute = AttributeImpl.of(useKey, value); - } - getAttributes().put(useKey, attribute); - return attribute; - } - - public Attribute addValue(Attribute attribute) { - return addValue(attribute.getName(), attribute.getValue()); - } - - public Attributes addValues(Attributes attributes) { - for (Attribute attribute : attributes.values()) { - addValue(attribute.getName(), attribute.getValue()); - } - return this; - } - - public Attribute addValue(CharSequence key, CharSequence value) { - Attribute attribute; - String useKey = String.valueOf(key); - if (attributes == null) { - attribute = AttributeImpl.of(key, value); - } else { - attribute = attributes.get(useKey); - if (attribute != null) attribute = attribute.setValue(value); - else attribute = AttributeImpl.of(useKey, value); - } - getAttributes().put(useKey, attribute); - return attribute; - } - - public Attribute removeValue(Attribute attribute) { - return removeValue(attribute.getName(), attribute.getValue()); - } - - public Attribute remove(Attribute attribute) { - return remove(attribute.getName()); - } - - public Attribute removeValue(CharSequence key, CharSequence value) { - if (attributes == null || key == null || key.length() == 0) return null; - - String useKey = String.valueOf(key); - Attribute oldAttribute = attributes.get(useKey); - Attribute attribute = oldAttribute.removeValue(value); - getAttributes().put(useKey, attribute); - return attribute; - } - public boolean contains(CharSequence key) { if (attributes == null || key == null || key.length() == 0) return false; @@ -127,12 +58,9 @@ public boolean isEmpty() { return attributes == null || attributes.isEmpty(); } - public void clear() { - attributes = null; - } - @SuppressWarnings("unchecked") public Set keySet() { + // CAUTION: attributes can be modified through this mutable set return attributes != null ? attributes.keySet() : Collections.EMPTY_SET; } @@ -143,6 +71,7 @@ public Collection values() { @SuppressWarnings("unchecked") public Set> entrySet() { + // CAUTION: attributes can be modified through this mutable set return attributes != null ? attributes.entrySet() : Collections.EMPTY_SET; } @@ -158,23 +87,6 @@ public int size() { return attributes == null ? 0 : attributes.size(); } - public Attribute remove(CharSequence key) { - if (attributes == null || key == null || key.length() == 0) return null; - - String useKey = String.valueOf(key); - Attribute oldAttribute = attributes.get(useKey); - attributes.remove(useKey); - return oldAttribute; - } - - public void replaceValues(Attributes attributes) { - if (this.attributes == null) { - this.attributes = new LinkedHashMap<>(attributes.attributes); - } else { - this.attributes.putAll(attributes.attributes); - } - } - @Override public String toString() { StringBuilder sb = new StringBuilder(); diff --git a/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/HtmlAppendableBase.java b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/HtmlAppendableBase.java index aa3ad225c..156bcf48d 100644 --- a/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/HtmlAppendableBase.java +++ b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/HtmlAppendableBase.java @@ -14,7 +14,7 @@ public class HtmlAppendableBase> implements HtmlAppendable { final private LineAppendable appendable; - private @Nullable Attributes currentAttributes; + private @Nullable MutableAttributes currentAttributes; private boolean indentOnFirstEol = false; private boolean lineOnChildText = false; private boolean withAttributes = false; @@ -133,7 +133,7 @@ public T text(@NotNull CharSequence s) { @Override public T attr(@NotNull CharSequence attrName, @NotNull CharSequence value) { if (currentAttributes == null) { - currentAttributes = new Attributes(); + currentAttributes = new MutableAttributes(); } currentAttributes.addValue(attrName, value); return (T) this; @@ -143,7 +143,7 @@ public T attr(@NotNull CharSequence attrName, @NotNull CharSequence value) { @Override public T attr(@NotNull Attribute... attribute) { if (currentAttributes == null) { - currentAttributes = new Attributes(); + currentAttributes = new MutableAttributes(); } for (Attribute attr : attribute) { currentAttributes.addValue(attr.getName(), attr.getValue()); @@ -156,7 +156,7 @@ public T attr(@NotNull Attribute... attribute) { public T attr(@NotNull Attributes attributes) { if (!attributes.isEmpty()) { if (currentAttributes == null) { - currentAttributes = new Attributes(attributes); + currentAttributes = new MutableAttributes(attributes); } else { currentAttributes.addValues(attributes); } @@ -179,7 +179,7 @@ public Attributes getAttributes() { @NotNull @Override public T setAttributes(@NotNull Attributes attributes) { - currentAttributes = attributes; + currentAttributes = attributes.toMutable(); return (T) this; } diff --git a/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/MutableAttributes.java b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/MutableAttributes.java new file mode 100644 index 000000000..7bd767647 --- /dev/null +++ b/flexmark-util-html/src/main/java/com/vladsch/flexmark/util/html/MutableAttributes.java @@ -0,0 +1,135 @@ +package com.vladsch.flexmark.util.html; + +import java.util.LinkedHashMap; + +public class MutableAttributes extends Attributes { + public MutableAttributes() { + super(); + } + + public MutableAttributes(Attributes attributes) { + super(attributes); + } + + @Override + public MutableAttributes toMutable() { + return this; + } + + @Override + public Attributes toImmutable() { + return new Attributes(this); + } + + protected LinkedHashMap getAttributes() { + if (attributes == null) { + attributes = new LinkedHashMap<>(); + } + return attributes; + } + + public Attribute replaceValue(Attribute attribute) { + return replaceValue(attribute.getName(), attribute.getValue()); + } + + /** + * Attribute dependent value replacement + * class and style append new values to existing ones + * others set it to the new value + * + * @param key attribute name + * @param value new value + * @return new attribute + */ + public Attribute replaceValue(CharSequence key, CharSequence value) { + String useKey = String.valueOf(key); + Attribute attribute; + if (attributes == null) { + attribute = AttributeImpl.of(useKey, value); + } else { + attribute = attributes.get(useKey); + if (attribute != null) attribute = attribute.replaceValue(value); + else attribute = AttributeImpl.of(useKey, value); + } + getAttributes().put(useKey, attribute); + return attribute; + } + + public Attribute addValue(Attribute attribute) { + return addValue(attribute.getName(), attribute.getValue()); + } + + public MutableAttributes addValues(Attributes attributes) { + for (Attribute attribute : attributes.values()) { + addValue(attribute.getName(), attribute.getValue()); + } + return this; + } + + public Attribute addValue(CharSequence key, CharSequence value) { + Attribute attribute; + String useKey = String.valueOf(key); + if (attributes == null) { + attribute = AttributeImpl.of(key, value); + } else { + attribute = attributes.get(useKey); + if (attribute != null) attribute = attribute.setValue(value); + else attribute = AttributeImpl.of(useKey, value); + } + getAttributes().put(useKey, attribute); + return attribute; + } + + public Attribute removeValue(Attribute attribute) { + return removeValue(attribute.getName(), attribute.getValue()); + } + + public Attribute remove(Attribute attribute) { + return remove(attribute.getName()); + } + + public Attribute removeValue(CharSequence key, CharSequence value) { + if (attributes == null || key == null || key.length() == 0) return null; + + String useKey = String.valueOf(key); + Attribute oldAttribute = attributes.get(useKey); + Attribute attribute = oldAttribute.removeValue(value); + getAttributes().put(useKey, attribute); + return attribute; + } + + public void clear() { + attributes = null; + } + + public Attribute remove(CharSequence key) { + if (attributes == null || key == null || key.length() == 0) return null; + + String useKey = String.valueOf(key); + Attribute oldAttribute = attributes.get(useKey); + attributes.remove(useKey); + return oldAttribute; + } + + public void replaceValues(MutableAttributes attributes) { + if (this.attributes == null) { + this.attributes = new LinkedHashMap<>(attributes.attributes); + } else { + this.attributes.putAll(attributes.attributes); + } + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder(); + String sep = ""; + for (String attrName : keySet()) { + sb.append(sep).append(attrName); + Attribute attribute = attributes.get(attrName); + if (!attribute.getValue().isEmpty()) sb.append("=").append("\"").append(attribute.getValue().replace("\"", "\\\"")).append("\""); + sep = " "; + } + + return "MutableAttributes{" + sb.toString() + '}'; + } +} diff --git a/flexmark-util-misc/flexmark-util-misc.iml b/flexmark-util-misc/flexmark-util-misc.iml index 59008a1cf..f8b782e6a 100644 --- a/flexmark-util-misc/flexmark-util-misc.iml +++ b/flexmark-util-misc/flexmark-util-misc.iml @@ -25,7 +25,7 @@ - + diff --git a/flexmark-util-misc/pom.xml b/flexmark-util-misc/pom.xml index 356e47741..6deea469f 100644 --- a/flexmark-util-misc/pom.xml +++ b/flexmark-util-misc/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-misc @@ -20,7 +20,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ArrayUtils.java b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ArrayUtils.java index 2c629f53c..a4d4159bc 100644 --- a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ArrayUtils.java +++ b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ArrayUtils.java @@ -3,6 +3,7 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.lang.reflect.Array; import java.util.BitSet; import java.util.Objects; import java.util.function.Predicate; @@ -12,6 +13,18 @@ public static boolean contained(T value, T[] array) { return indexOf(value, array) != -1; } + @SafeVarargs + public static T[] append(Class elemClass, T[] array, T... values) { + if (values.length > 0) { + //noinspection unchecked + T[] newInstance = (T[]) Array.newInstance(elemClass, array.length + values.length); + System.arraycopy(array, 0, newInstance, 0, array.length); + System.arraycopy(values, 0, newInstance, array.length, values.length); + return newInstance; + } + return array; + } + public static boolean contained(int value, int[] array) { for (int item : array) { if (item == value) return true; @@ -54,6 +67,7 @@ public static int indexOf(T t, T[] ts, int fromIndex, int endIndex) { * If it is negative, it has the same effect as if it were 0: -1 is returned. * @param predicate condition for matching the search * @param type of array + * * @return the index of the next occurrence of a match in the array which is * greater than or equal to {@code fromIndex}, or {@code -1} * if match does not occur after that point. @@ -109,6 +123,7 @@ public static int lastIndexOf(T t, T[] ts, int startIndex, int fromIndex) { * -1 is returned. * @param predicate condition for matching the search * @param type of array + * * @return the index of the last occurrence of a match in the array which is * less than or equal to {@code fromIndex}, or {@code -1} * if match does not occur before that point. diff --git a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/BitFieldSet.java b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/BitFieldSet.java index 22dd8cf63..62fb72ca3 100644 --- a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/BitFieldSet.java +++ b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/BitFieldSet.java @@ -3,7 +3,6 @@ import org.jetbrains.annotations.NotNull; import java.io.Serializable; -import java.lang.reflect.Field; import java.util.AbstractSet; import java.util.Collection; import java.util.Iterator; @@ -36,26 +35,19 @@ static class UniverseLoader { @SuppressWarnings("rawtypes") @NotNull - public static Enum[] getUniverseSlow(Class elementType) { + public static > Enum[] getUniverseSlow(Class elementType) { assert (elementType.isEnum()); Enum[] cachedUniverse = enumUniverseMap.get(elementType); if (cachedUniverse != null) return cachedUniverse; - Field[] fields = elementType.getFields(); - int enums = 0; - for (Field field : fields) { - if (field.getType().isEnum()) enums++; - } - + E[] constants = elementType.getEnumConstants(); + int enums = constants.length; if (enums > 0) { cachedUniverse = new Enum[enums]; enums = 0; - for (Field field : fields) { - if (field.getType().isEnum()) { - //noinspection unchecked - cachedUniverse[enums++] = Enum.valueOf((Class) field.getType(), field.getName()); - } + for (E constant : constants) { + cachedUniverse[enums++] = constant; } } else { cachedUniverse = ZERO_LENGTH_ENUM_ARRAY; @@ -621,19 +613,17 @@ public int size() { } /** - * Returns true if this set contains no elements. - * - * @return true if this set contains no elements + * @return true if this set contains no elements */ public boolean isEmpty() { return elements == 0; } /** - * Returns true if this set contains the specified element. + * Returns true if this set contains the specified element. * * @param e element to be checked for containment in this collection - * @return true if this set contains the specified element + * @return true if this set contains the specified element */ public boolean contains(Object e) { if (e == null) @@ -651,8 +641,8 @@ public boolean contains(Object e) { * Adds the specified element to this set if it is not already present. * * @param e element to be added to this set - * @return true if the set changed as a result of the call - * @throws NullPointerException if e is null + * @return true if the set changed as a result of the call + * @throws NullPointerException if e is null */ public boolean add(E e) { typeCheck(e); @@ -666,7 +656,7 @@ public boolean add(E e) { * Removes the specified element from this set if it is present. * * @param e element to be removed from this set, if present - * @return true if the set contained the specified element + * @return true if the set contained the specified element */ public boolean remove(Object e) { if (e == null) @@ -683,11 +673,11 @@ public boolean remove(Object e) { // Bulk Operations /** - * Returns true if this set contains all of the elements + * Returns true if this set contains all of the elements * in the specified collection. * * @param c collection to be checked for containment in this set - * @return true if this set contains all of the elements + * @return true if this set contains all of the elements * in the specified collection * @throws NullPointerException if the specified collection is null */ @@ -706,7 +696,7 @@ public boolean containsAll(Collection c) { * Adds all of the elements in the specified collection to this set. * * @param c collection whose elements are to be added to this set - * @return true if this set changed as a result of the call + * @return true if this set changed as a result of the call * @throws NullPointerException if the specified collection or any * of its elements are null */ @@ -733,7 +723,7 @@ public boolean addAll(Collection c) { * the specified collection. * * @param c elements to be removed from this set - * @return true if this set changed as a result of the call + * @return true if this set changed as a result of the call * @throws NullPointerException if the specified collection is null */ public boolean removeAll(Collection c) { @@ -754,7 +744,7 @@ public boolean removeAll(Collection c) { * specified collection. * * @param c elements to be retained in this set - * @return true if this set changed as a result of the call + * @return true if this set changed as a result of the call * @throws NullPointerException if the specified collection is null */ public boolean retainAll(Collection c) { @@ -870,12 +860,12 @@ private void readObject(java.io.ObjectInputStream stream) /** * Compares the specified object with this set for equality. Returns - * true if the given object is also a set, the two sets have + * true if the given object is also a set, the two sets have * the same size, and every member of the given set is contained in * this set. * * @param o object to be compared for equality with this set - * @return true if the specified object is equal to this set + * @return true if the specified object is equal to this set */ public boolean equals(Object o) { if (!(o instanceof BitFieldSet)) @@ -894,7 +884,7 @@ public boolean equals(Object o) { * @param elementType the class object of the element type for this enum * set * @return An empty enum set of the specified type. - * @throws NullPointerException if elementType is null + * @throws NullPointerException if elementType is null */ public static > BitFieldSet noneOf(Class elementType) { if (!elementType.isEnum()) @@ -1053,7 +1043,7 @@ public static > byte getBitField(byte elements, E e1) { * @param elementType the class object of the element type for this enum * set * @return An enum set containing all the elements in the specified type. - * @throws NullPointerException if elementType is null + * @throws NullPointerException if elementType is null */ public static > BitFieldSet allOf(Class elementType) { BitFieldSet result = noneOf(elementType); @@ -1068,7 +1058,7 @@ public static > BitFieldSet allOf(Class elementType) { * @param The class of the elements in the set * @param s the enum set from which to initialize this enum set * @return A copy of the specified enum set. - * @throws NullPointerException if s is null + * @throws NullPointerException if s is null */ public static > BitFieldSet copyOf(BitFieldSet s) { return s.clone(); @@ -1076,7 +1066,7 @@ public static > BitFieldSet copyOf(BitFieldSet s) { /** * Creates an enum set initialized from the specified collection. If - * the specified collection is an BitFieldSet instance, this static + * the specified collection is an BitFieldSet instance, this static * factory method behaves identically to {@link #copyOf(BitFieldSet)}. * Otherwise, the specified collection must contain at least one element * (in order to determine the new enum set's element type). @@ -1084,9 +1074,9 @@ public static > BitFieldSet copyOf(BitFieldSet s) { * @param The class of the elements in the collection * @param c the collection from which to initialize this enum set * @return An enum set initialized from the given collection. - * @throws IllegalArgumentException if c is not an - * BitFieldSet instance and contains no elements - * @throws NullPointerException if c is null + * @throws IllegalArgumentException if c is not an + * BitFieldSet instance and contains no elements + * @throws NullPointerException if c is null */ public static > BitFieldSet copyOf(Collection c) { if (c instanceof BitFieldSet) { @@ -1110,7 +1100,7 @@ public static > BitFieldSet copyOf(Collection c) { * @param The class of the elements in the enum set * @param s the enum set from whose complement to initialize this enum set * @return The complement of the specified set in this set - * @throws NullPointerException if s is null + * @throws NullPointerException if s is null */ public static > BitFieldSet complementOf(BitFieldSet s) { BitFieldSet result = copyOf(s); @@ -1130,7 +1120,7 @@ public static > BitFieldSet complementOf(BitFieldSet s) * @param The class of the specified element and of the set * @param e the element that this set is to contain initially * @return an enum set initially containing the specified element - * @throws NullPointerException if e is null + * @throws NullPointerException if e is null */ public static > BitFieldSet of(E e) { BitFieldSet result = noneOf(e.getDeclaringClass()); @@ -1253,7 +1243,7 @@ public static > BitFieldSet of( * @param rest the remaining elements the set is to contain initially * @return an enum set initially containing the specified elements * @throws NullPointerException if any of the specified elements are null, - * or if rest is null + * or if rest is null */ @SafeVarargs public static > BitFieldSet of(E first, E... rest) { @@ -1275,7 +1265,7 @@ public static > BitFieldSet of(E first, E... rest) { * @param rest the remaining elements the set is to contain initially * @return an enum set initially containing the specified elements * @throws NullPointerException if any of the specified elements are null, - * or if rest is null + * or if rest is null */ public static > BitFieldSet of(@NotNull Class declaringClass, E[] rest) { BitFieldSet result = noneOf(declaringClass); diff --git a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/CharPredicate.java b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/CharPredicate.java index 0937bbd61..f89d34841 100644 --- a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/CharPredicate.java +++ b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/CharPredicate.java @@ -25,13 +25,17 @@ public interface CharPredicate extends IntPredicate { CharPredicate SPACE_TAB = value -> value == ' ' || value == '\t'; CharPredicate SPACE_TAB_NUL = value -> value == ' ' || value == '\t' || value == '\0'; CharPredicate SPACE_TAB_LINE_SEP = value -> value == ' ' || value == '\t' || value == '\u2028'; + CharPredicate SPACE_TAB_NBSP_LINE_SEP = value -> value == ' ' || value == '\t' || value == '\u00A0' || value == '\u2028'; CharPredicate SPACE_EOL = value -> value == ' ' || value == '\n'; CharPredicate SPACE_ANY_EOL = value -> value == ' ' || value == '\r' || value == '\n'; CharPredicate SPACE_TAB_NBSP = value -> value == ' ' || value == '\t' || value == '\u00A0'; CharPredicate SPACE_TAB_EOL = value -> value == ' ' || value == '\t' || value == '\n'; + CharPredicate SPACE_TAB_NBSP_EOL = value -> value == ' ' || value == '\t' || value == '\n' || value == '\u00A0'; CharPredicate WHITESPACE = value -> value == ' ' || value == '\t' || value == '\n' || value == '\r'; CharPredicate WHITESPACE_OR_NUL = value -> value == ' ' || value == '\t' || value == '\n' || value == '\r' || value == '\0'; CharPredicate WHITESPACE_NBSP = value -> value == ' ' || value == '\t' || value == '\n' || value == '\r' || value == '\u00A0'; + CharPredicate WHITESPACE_NBSP_OR_NUL = value -> value == ' ' || value == '\t' || value == '\n' || value == '\r' || value == '\u00A0' || value == '\0'; + CharPredicate BLANKSPACE = value -> value == ' ' || value == '\t' || value == '\n' || value == '\r' || value == '\u000B' || value == '\f'; CharPredicate HEXADECIMAL_DIGITS = value -> value >= '0' && value <= '9' || value >= 'a' && value <= 'f' || value >= 'A' && value <= 'F'; CharPredicate DECIMAL_DIGITS = value -> value >= '0' && value <= '9'; CharPredicate OCTAL_DIGITS = value -> value >= '0' && value <= '7'; diff --git a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/FileUtil.java b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/FileUtil.java index 40518c28a..547926a2a 100644 --- a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/FileUtil.java +++ b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/FileUtil.java @@ -1,7 +1,11 @@ package com.vladsch.flexmark.util.misc; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + import java.io.*; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; import static com.vladsch.flexmark.util.misc.Utils.suffixWith; @@ -32,10 +36,11 @@ public static File plus(File receiver, String name) { return new File(receiver, name); } + @Nullable public static String getFileContent(File receiver) { StringBuilder sb = new StringBuilder(); + String line; try { - String line; InputStream inputStream = new FileInputStream(receiver); InputStreamReader streamReader = new InputStreamReader(inputStream, StandardCharsets.UTF_8); BufferedReader reader = new BufferedReader(streamReader); @@ -48,7 +53,40 @@ public static String getFileContent(File receiver) { inputStream.close(); return sb.toString(); } catch (IOException e) { - throw new RuntimeException(e); + e.printStackTrace(); + } + return null; + } + + @NotNull + public static String getFileContentWithExceptions(File receiver) throws IOException { + StringBuilder sb = new StringBuilder(); + String line; + InputStream inputStream = new FileInputStream(receiver); + InputStreamReader streamReader = new InputStreamReader(inputStream, StandardCharsets.UTF_8); + BufferedReader reader = new BufferedReader(streamReader); + while ((line = reader.readLine()) != null) { + sb.append(line); + sb.append("\n"); + } + reader.close(); + streamReader.close(); + inputStream.close(); + return sb.toString(); + } + + @Nullable + public static byte[] getFileContentBytes(File receiver) { + try { + return Files.readAllBytes(receiver.toPath()); + } catch (IOException e) { + e.printStackTrace(); } + return null; + } + + @NotNull + public static byte[] getFileContentBytesWithExceptions(File receiver) throws IOException { + return Files.readAllBytes(receiver.toPath()); } } diff --git a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ImageUtils.java b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ImageUtils.java index 2cca4c6ca..0c364cbf1 100644 --- a/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ImageUtils.java +++ b/flexmark-util-misc/src/main/java/com/vladsch/flexmark/util/misc/ImageUtils.java @@ -163,6 +163,35 @@ public static BufferedImage loadImageFromFile(File cachedImageFile) { return null; } + /** + * @param cachedImageBytes file + * @param idPath image file path for error reporting + * @return Could be {@code null} if the image could not be read from the file (because of whatever strange + * reason). + */ + public static BufferedImage loadImageFromContent(byte[] cachedImageBytes, String idPath) { + if (cachedImageBytes == null) { + return null; + } + + ByteArrayInputStream inputStream = new ByteArrayInputStream(cachedImageBytes); + + try { + return ImageIO.read(inputStream); + } catch (IndexOutOfBoundsException | IOException e) { + System.err.print("*"); + System.err.println("could not read from image bytes for " + idPath); + } finally { + try { + inputStream.close(); + } catch (IOException e) { + e.printStackTrace(); + } + } + + return null; + } + public static String base64Encode(BufferedImage image) { String imageString = null; ByteArrayOutputStream bos = new ByteArrayOutputStream(); diff --git a/flexmark-util-options/flexmark-util-options.iml b/flexmark-util-options/flexmark-util-options.iml index bbe6bc2bc..20e685d54 100644 --- a/flexmark-util-options/flexmark-util-options.iml +++ b/flexmark-util-options/flexmark-util-options.iml @@ -25,7 +25,7 @@ - + diff --git a/flexmark-util-options/pom.xml b/flexmark-util-options/pom.xml index 7a6ab4966..444be4a17 100644 --- a/flexmark-util-options/pom.xml +++ b/flexmark-util-options/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-options @@ -28,7 +28,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-sequence/flexmark-util-sequence.iml b/flexmark-util-sequence/flexmark-util-sequence.iml index de4a99d9c..af7cf51b1 100644 --- a/flexmark-util-sequence/flexmark-util-sequence.iml +++ b/flexmark-util-sequence/flexmark-util-sequence.iml @@ -21,15 +21,16 @@ + - + - - + + \ No newline at end of file diff --git a/flexmark-util-sequence/pom.xml b/flexmark-util-sequence/pom.xml index 4927660a5..38e7e4e34 100644 --- a/flexmark-util-sequence/pom.xml +++ b/flexmark-util-sequence/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-sequence @@ -32,7 +32,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequence.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequence.java index c14287b04..0dabaef55 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequence.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequence.java @@ -40,18 +40,6 @@ static BasedSequence of(@Nullable CharSequence charSequence) { return BasedSequenceImpl.create(charSequence); } - @NotNull - @Deprecated - static BasedSequence of(@Nullable CharSequence charSequence, int startIndex) { - return of(charSequence).subSequence(startIndex); - } - - @NotNull - @Deprecated - static BasedSequence of(@Nullable CharSequence charSequence, int startIndex, int endIndex) { - return of(charSequence).subSequence(startIndex, endIndex); - } - @NotNull static BasedSequence ofSpaces(int count) { return of(RepeatedSequence.ofSpaces(count)); @@ -340,6 +328,37 @@ static BasedSequence repeatOf(@NotNull CharSequence chars, int startIndex, int e @NotNull BasedSequence extendByOneOfAny(@NotNull CharPredicate charSet); + /** + * Test for line containing some of the characters in the set + * + * @param charSet set of characters to be tested + * @return true if line contains some characters in the set + */ + boolean containsSomeIn(@NotNull CharPredicate charSet); + + /** + * Test for line containing some characters not in the set + * + * @param charSet set of characters to be tested + * @return true if line contains some characters not in the set + */ + boolean containsSomeNotIn(@NotNull CharPredicate charSet); + /** + * Test for line contains only characters from the set + * + * @param charSet set of characters to be tested + * @return true if line contains some characters from the set + */ + boolean containsOnlyIn(@NotNull CharPredicate charSet); + + /** + * Test for line containing only characters not in the set + * + * @param charSet set of characters to be tested + * @return true if line contains some characters from the set + */ + boolean containsOnlyNotIn(@NotNull CharPredicate charSet); + /** * Extend this based sequence to include characters from underlying based sequence not in character set * diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequenceImpl.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequenceImpl.java index e5881b684..2a77d99c1 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequenceImpl.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/BasedSequenceImpl.java @@ -42,7 +42,7 @@ public BasedSequence nullSequence() { @NotNull @Override public BasedSequence sequenceOf(@Nullable CharSequence charSequence, int startIndex, int endIndex) { - return of(charSequence).subSequence(startIndex, endIndex); + return BasedSequence.of(charSequence).subSequence(startIndex, endIndex); } @SuppressWarnings("unchecked") @@ -195,7 +195,35 @@ public BasedSequence intersect(@NotNull BasedSequence other) { else return this.baseSubSequence(Utils.max(getStartOffset(), other.getStartOffset()), Utils.min(getEndOffset(), other.getEndOffset())); } -// @formatter:off + @Override + public boolean containsSomeIn(@NotNull CharPredicate charSet) { + int iMax = length(); + for (int i = 0; i < iMax; i++) { + if (charSet.test(charAt(i))) return true; + } + return false; + } + + @Override + public boolean containsSomeNotIn(@NotNull CharPredicate charSet) { + int iMax = length(); + for (int i = 0; i < iMax; i++) { + if (!charSet.test(charAt(i))) return true; + } + return false; + } + + @Override + public boolean containsOnlyIn(@NotNull CharPredicate charSet) { + return !containsSomeNotIn(charSet); + } + + @Override + public boolean containsOnlyNotIn(@NotNull CharPredicate charSet) { + return !containsSomeIn(charSet); + } + + // @formatter:off @NotNull @Override public BasedSequence extendByAny(@NotNull CharPredicate charSet) { return extendByAny(charSet, Integer.MAX_VALUE - getEndOffset()); } @NotNull @Override public BasedSequence extendByOneOfAny(@NotNull CharPredicate charSet) { return extendByAny(charSet, 1); } @@ -386,19 +414,4 @@ else if (charSequence instanceof BasedSequence) { return SubSequence.create(charSequence); } } - - @Deprecated - public static BasedSequence of(@Nullable CharSequence charSequence) { - return BasedSequence.of(charSequence); - } - - @Deprecated - public static BasedSequence of(@Nullable CharSequence charSequence, int startIndex) { - return BasedSequence.of(charSequence).subSequence(startIndex); - } - - @Deprecated - public static BasedSequence of(@Nullable CharSequence charSequence, int startIndex, int endIndex) { - return BasedSequence.of(charSequence).subSequence(startIndex, endIndex); - } } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/CharSubSequence.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/CharSubSequence.java index f82acc9c1..a6ba71c3c 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/CharSubSequence.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/CharSubSequence.java @@ -152,10 +152,8 @@ public static CharSubSequence of(char[] chars, int startIndex, int endIndex) { * @param startIndex start index in sequence * @param endIndex end index in sequence * @return char based sequence - * @deprecated NOTE: use BasedSequence.of() for creating based sequences */ - @Deprecated - public static CharSubSequence of(CharSequence charSequence, int startIndex, int endIndex) { + private static CharSubSequence of(CharSequence charSequence, int startIndex, int endIndex) { assert startIndex >= 0 && startIndex <= endIndex && endIndex <= charSequence.length(); CharSubSequence charSubSequence; diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Escaping.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Escaping.java index 8fb5fc981..e519884bb 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Escaping.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Escaping.java @@ -6,6 +6,7 @@ import org.jetbrains.annotations.Nullable; import java.nio.charset.StandardCharsets; +import java.util.List; import java.util.Random; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -63,32 +64,44 @@ public class Escaping { final private static Replacer UNSAFE_CHAR_REPLACER = new Replacer() { @Override public void replace(@NotNull String s, @NotNull StringBuilder sb) { - if (s.equals("&")) { - sb.append("&"); - } else if (s.equals("<")) { - sb.append("<"); - } else if (s.equals(">")) { - sb.append(">"); - } else if (s.equals("\"")) { - sb.append("""); - } else { - sb.append(s); + switch (s) { + case "&": + sb.append("&"); + break; + case "<": + sb.append("<"); + break; + case ">": + sb.append(">"); + break; + case "\"": + sb.append("""); + break; + default: + sb.append(s); + break; } } @Override public void replace(@NotNull BasedSequence original, int startIndex, int endIndex, @NotNull ReplacedTextMapper textMapper) { String s1 = original.subSequence(startIndex, endIndex).toString(); - if (s1.equals("&")) { - textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf("&", BasedSequence.NULL)); - } else if (s1.equals("<")) { - textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf("<", BasedSequence.NULL)); - } else if (s1.equals(">")) { - textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf(">", BasedSequence.NULL)); - } else if (s1.equals("\"")) { - textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf(""", BasedSequence.NULL)); - } else { - textMapper.addOriginalText(startIndex, endIndex); + switch (s1) { + case "&": + textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf("&", BasedSequence.NULL)); + break; + case "<": + textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf("<", BasedSequence.NULL)); + break; + case ">": + textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf(">", BasedSequence.NULL)); + break; + case "\"": + textMapper.addReplacedText(startIndex, endIndex, PrefixedSubSequence.prefixOf(""", BasedSequence.NULL)); + break; + default: + textMapper.addOriginalText(startIndex, endIndex); + break; } } }; @@ -331,6 +344,23 @@ public static BasedSequence unescapeHtml(@NotNull BasedSequence s, @NotNull Repl } } + /** + * Replace entities and backslash escapes with literal characters. + * + * @param s based sequence to un-escape + * @param textMapper replaced text mapper to update for the changed text + * @return un-escaped sequence + */ + @NotNull + public static BasedSequence unescapeHtml(@NotNull BasedSequence s, @NotNull List ranges, @NotNull ReplacedTextMapper textMapper) { + int indexOfAny = s.indexOf('&'); + if (indexOfAny != -1) { + return replaceAll(ENTITY_ONLY, s, ranges, ENTITY_REPLACER, textMapper); + } else { + return s; + } + } + /** * Normalize eol: embedded \r and \r\n are converted to \n * @@ -478,7 +508,7 @@ public static String percentEncodeUrl(@NotNull CharSequence s) { } /** - * @param s string to encode + * @param s string to encode * @param textMapper text mapper to update for the replaced text * @return encoded string */ @@ -497,7 +527,7 @@ public static String percentDecodeUrl(@NotNull CharSequence s) { } /** - * @param s string to encode + * @param s string to encode * @param textMapper text mapper to update for the replaced text * @return encoded string */ @@ -645,8 +675,15 @@ private static String replaceAll(@NotNull Pattern p, @NotNull CharSequence s, @N } @NotNull - private static BasedSequence replaceAll(Pattern p, @NotNull BasedSequence s, @NotNull Replacer replacer, @NotNull ReplacedTextMapper textMapper) { + private static BasedSequence replaceAll(@NotNull Pattern p, @NotNull BasedSequence s, @NotNull Replacer replacer, @NotNull ReplacedTextMapper textMapper) { + return replaceAll(p, s, 0, s.length(), replacer, textMapper); + } + + @NotNull + private static BasedSequence replaceAll(@NotNull Pattern p, @NotNull BasedSequence s, int startOffset, int endOffset, @NotNull Replacer replacer, @NotNull ReplacedTextMapper textMapper) { Matcher matcher = p.matcher(s); + matcher.region(startOffset, endOffset); + matcher.useTransparentBounds(false); if (textMapper.isModified()) { textMapper.startNestedReplacement(s); @@ -664,7 +701,37 @@ private static BasedSequence replaceAll(Pattern p, @NotNull BasedSequence s, @No lastEnd = matcher.end(); } while (matcher.find()); - if (lastEnd != s.length()) { + if (lastEnd < s.length()) { + textMapper.addOriginalText(lastEnd, s.length()); + } + + return textMapper.getReplacedSequence(); + } + + @NotNull + private static BasedSequence replaceAll(@NotNull Pattern p, @NotNull BasedSequence s, @NotNull List ranges, @NotNull Replacer replacer, @NotNull ReplacedTextMapper textMapper) { + Matcher matcher = p.matcher(s); + matcher.useTransparentBounds(false); + + if (textMapper.isModified()) { + textMapper.startNestedReplacement(s); + } + + int lastEnd = 0; + + for (Range range : ranges) { + int start = Utils.rangeLimit(range.getStart(), lastEnd, s.length()); + int end = Utils.rangeLimit(range.getEnd(), start, s.length()); + matcher.region(start, end); + + while (matcher.find()) { + textMapper.addOriginalText(lastEnd, matcher.start()); + replacer.replace(s, matcher.start(), matcher.end(), textMapper); + lastEnd = matcher.end(); + } + } + + if (lastEnd < s.length()) { textMapper.addOriginalText(lastEnd, s.length()); } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Html5Entities.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Html5Entities.java index bde122f14..6394f093e 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Html5Entities.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/Html5Entities.java @@ -16,7 +16,7 @@ public class Html5Entities { final private static Map NAMED_CHARACTER_REFERENCES = readEntities(); final private static Pattern NUMERIC_PATTERN = Pattern.compile("^&#[Xx]?"); - final private static String ENTITY_PATH = "/com/vladsch/flexmark/util/html/entities.properties"; + final private static String ENTITY_PATH = "/com/vladsch/flexmark/util/sequence/entities.properties"; public static String entityToString(String input) { Matcher matcher = NUMERIC_PATTERN.matcher(input); diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequence.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequence.java index 216b44fc6..b3f1797e1 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequence.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequence.java @@ -7,6 +7,7 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.util.Collection; import java.util.List; import java.util.function.BiPredicate; import java.util.function.Predicate; @@ -1403,4 +1404,7 @@ default int getColumnAtIndex(int index) { */ @Nullable String toStringOrNull(); + + boolean isIn(@NotNull String[] texts); + boolean isIn(@NotNull Collection texts); } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequenceBase.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequenceBase.java index fee6a5dd2..9e30ec812 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequenceBase.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/IRichSequenceBase.java @@ -11,6 +11,7 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.util.Collection; import java.util.List; import java.util.function.BiPredicate; import java.util.function.Predicate; @@ -903,4 +904,14 @@ final public T extractRanges(Iterable ranges) { @Override final public int columnAtIndex(int index) { return SequenceUtils.columnAtIndex(this, index);} @NotNull @Override final public Pair lineColumnAtIndex(int index) {return SequenceUtils.lineColumnAtIndex(this, index);} // @formatter:on + + @Override + public boolean isIn(@NotNull String[] texts) { + return SequenceUtils.containedBy(texts, this); + } + + @Override + public boolean isIn(@NotNull Collection texts) { + return SequenceUtils.containedBy(texts, this); + } } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/LineAppendableImpl.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/LineAppendableImpl.java index 56e696377..4dd591aa3 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/LineAppendableImpl.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/LineAppendableImpl.java @@ -319,13 +319,13 @@ private LineInfo getLineRange(int start, int end, CharSequence prefix) { CharSequence sequence = appendable.toSequence(); CharSequence eol = trimmedEOL(sequence); - + if (eol == null || eol.length() == 0) { eol = SequenceUtils.EOL; } // KLUDGE: end always has 1 EOL character removed, however, if there is a \r before \n then one more char needs to be removed from end of text - CharSequence text = start == Range.NULL.getStart() && end == Range.NULL.getEnd() ? BasedSequence.NULL + CharSequence text = start == Range.NULL.getStart() && end == Range.NULL.getEnd() ? BasedSequence.NULL : sequence.subSequence(start, Math.max(start, end - Math.max(0, eol.length() - 1))); if (start >= end) { @@ -849,7 +849,7 @@ public T appendToNoLine(@NotNull T out, boolean withPrefi // NOTE: these are tail blank lines if (consecutiveBlankLines < maxTrailingBlankLines) { consecutiveBlankLines++; - if (withPrefixes) out.append(trimEnd(info.getPrefix())); + if (withPrefixes) out.append(isTrimTrailingWhitespace() ? trimEnd(info.getPrefix()) : info.getPrefix()); if (notDanglingLine && (tailEOL || consecutiveBlankLines != maxTrailingBlankLines)) { out.append(EOL); } @@ -857,7 +857,7 @@ public T appendToNoLine(@NotNull T out, boolean withPrefi } else { if (consecutiveBlankLines < maxBlankLines) { consecutiveBlankLines++; - if (withPrefixes) out.append(trimEnd(info.getPrefix())); + if (withPrefixes) out.append(isTrimTrailingWhitespace() ? trimEnd(info.getPrefix()) : info.getPrefix()); if (notDanglingLine) out.append(EOL); } } @@ -998,8 +998,8 @@ public void setPrefixLength(int lineIndex, int prefixLength) { LineInfo info = lines.get(lineIndex); CharSequence line = info.lineSeq; - if (prefixLength < 0 || prefixLength >= line.length() - 1) - throw new IllegalArgumentException(String.format("prefixLength %d is out of valid range [0, %d) for the line", prefixLength, line.length() - 1)); + if (prefixLength < 0 || prefixLength > info.getTextEnd()) + throw new IllegalArgumentException(String.format("prefixLength %d is out of valid range [0, %d) for the line", prefixLength, info.getTextEnd() + 1)); if (prefixLength != info.prefixLength) { CharSequence prefix = line.subSequence(0, prefixLength); diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/SequenceUtils.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/SequenceUtils.java index d7b5ba2fd..d89e20c78 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/SequenceUtils.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/SequenceUtils.java @@ -8,10 +8,7 @@ import java.text.NumberFormat; import java.text.ParsePosition; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.function.Predicate; import static com.vladsch.flexmark.util.misc.Utils.rangeLimit; @@ -300,7 +297,7 @@ static int lastIndexOfAny(@NotNull CharSequence thizz, @NotNull CharPredicate s, * resorting to content comparison only if length and hashCodes are equal * * @param thizz char sequence to test for equality - * @param o any character sequence + * @param o any character sequence * @return true if character contents are equal */ @Contract(pure = true, value = "_, null -> false") @@ -1296,4 +1293,18 @@ static Pair parseNumberPrefixOrNull(@Nullable String text, @Null return null; } + + static boolean containedBy(@NotNull T[] items, @NotNull CharSequence element) { + for (T item : items) { + if (equals(element, item)) return true; + } + return false; + } + + static boolean containedBy(@NotNull Collection items, @NotNull CharSequence element) { + for (CharSequence item : items) { + if (equals(element, item)) return true; + } + return false; + } } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/SequenceBuilder.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/SequenceBuilder.java index 3e8949d65..5e1827cd8 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/SequenceBuilder.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/SequenceBuilder.java @@ -337,14 +337,6 @@ public String toString() { BasedSequence s = baseSeq.subSequence(((Range) part).getStart(), ((Range) part).getEnd()); if (s.isNotEmpty()) { - if (last != null && last.isNotEmpty() && last.getEndOffset() < s.getStartOffset() - && (BasedSequence.WHITESPACE.indexOf(last.charAt(last.length() - 1)) == -1) - && BasedSequence.WHITESPACE.indexOf(s.charAt(0)) == -1 - && s.baseSubSequence(last.getEndOffset(), s.getStartOffset()).endsWith(" ") - ) { - sb.append(' '); - } - s.appendTo(sb); } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/tree/SegmentTree.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/tree/SegmentTree.java index 99deff54b..41b5b3b47 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/tree/SegmentTree.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/builder/tree/SegmentTree.java @@ -8,6 +8,8 @@ import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.util.Arrays; + import static com.vladsch.flexmark.util.sequence.builder.tree.Segment.SegType.ANCHOR; import static com.vladsch.flexmark.util.sequence.builder.tree.Segment.SegType.BASE; @@ -290,7 +292,9 @@ public void addSegments(@NotNull IBasedSegmentBuilder builder, int startIndex builder.appendAnchor(startOffset); } + int currentEnd = startOffset; BasedSequence baseSequence = builder.getBaseSequence(); + for (int i = startPos; i < endPos; i++) { Segment segment = getSegment(i, baseSequence); @@ -316,12 +320,14 @@ public void addSegments(@NotNull IBasedSegmentBuilder builder, int startIndex } } else { assert charSequence instanceof BasedSequence; - builder.append(((BasedSequence) charSequence).getStartOffset(), ((BasedSequence) charSequence).getEndOffset()); + BasedSequence basedSequence = (BasedSequence) charSequence; + currentEnd = Math.max(currentEnd, basedSequence.getEndOffset()); + builder.append(basedSequence.getStartOffset(), basedSequence.getEndOffset()); } } if (endOffset != -1) { - builder.appendAnchor(endOffset); + builder.appendAnchor(Math.max(currentEnd, endOffset)); } } @@ -438,6 +444,9 @@ public static SegmentTreePos findSegmentPos(int index, int[] treeData, int start int iterations = 0; while (startPos < endPos) { int pos = (startPos + endPos) >> 1; + int lastStart = startPos; + int lastEnd = endPos; + iterations++; // System.out.println(String.format("Iteration[%d] pos: %d, [%d, %d)", iterations, pos, startPos, endPos)); @@ -453,6 +462,10 @@ public static SegmentTreePos findSegmentPos(int index, int[] treeData, int start return new SegmentTreePos(pos, startIndex, iterations); } } + + assert lastStart != startPos || lastEnd != endPos : "Range and position did not change after iteration: pos=" + pos + ", startPos=" + startPos + ", endPos=" + endPos + + "\n" + Arrays.toString(treeData) + ; } return null; } diff --git a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/mappers/SpaceMapper.java b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/mappers/SpaceMapper.java index 7dbf73b6c..441cc26fd 100644 --- a/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/mappers/SpaceMapper.java +++ b/flexmark-util-sequence/src/main/java/com/vladsch/flexmark/util/sequence/mappers/SpaceMapper.java @@ -7,6 +7,10 @@ public class SpaceMapper { final public static CharMapper toNonBreakSpace = new ToNonBreakSpace(); final public static CharMapper fromNonBreakSpace = new FromNonBreakSpace(); + + public static boolean areEquivalent(char c1, char c2) { + return c1 == c2 || (c1 == ' ' && c2 == SequenceUtils.NBSP) || (c2 == ' ' && c1 == SequenceUtils.NBSP); + } public static @NotNull CharMapper toSpaces(@NotNull CharPredicate predicate) { return new FromPredicate(predicate); diff --git a/flexmark-util-html/src/main/resources/com/vladsch/flexmark/util/html/entities.properties b/flexmark-util-sequence/src/main/resources/com/vladsch/flexmark/util/sequence/entities.properties similarity index 100% rename from flexmark-util-html/src/main/resources/com/vladsch/flexmark/util/html/entities.properties rename to flexmark-util-sequence/src/main/resources/com/vladsch/flexmark/util/sequence/entities.properties diff --git a/flexmark-util-visitor/flexmark-util-visitor.iml b/flexmark-util-visitor/flexmark-util-visitor.iml index cd849bc06..8abe26861 100644 --- a/flexmark-util-visitor/flexmark-util-visitor.iml +++ b/flexmark-util-visitor/flexmark-util-visitor.iml @@ -25,7 +25,7 @@ - + diff --git a/flexmark-util-visitor/pom.xml b/flexmark-util-visitor/pom.xml index 5fef59891..7bbbeace1 100644 --- a/flexmark-util-visitor/pom.xml +++ b/flexmark-util-visitor/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util-visitor @@ -20,7 +20,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util/flexmark-util.iml b/flexmark-util/flexmark-util.iml index 465b44780..7c2414280 100644 --- a/flexmark-util/flexmark-util.iml +++ b/flexmark-util/flexmark-util.iml @@ -24,7 +24,7 @@ - + @@ -39,4 +39,4 @@ - + \ No newline at end of file diff --git a/flexmark-util/pom.xml b/flexmark-util/pom.xml index f28f8cb44..3803b14a8 100644 --- a/flexmark-util/pom.xml +++ b/flexmark-util/pom.xml @@ -5,7 +5,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-util @@ -76,7 +76,7 @@ org.jetbrains annotations - 15.0 + 24.0.1 diff --git a/flexmark-util/src/test/java/com/vladsch/flexmark/util/ExceptionMatcher.java b/flexmark-util/src/test/java/com/vladsch/flexmark/util/ExceptionMatcher.java index 82a91b5ee..f8f71616e 100644 --- a/flexmark-util/src/test/java/com/vladsch/flexmark/util/ExceptionMatcher.java +++ b/flexmark-util/src/test/java/com/vladsch/flexmark/util/ExceptionMatcher.java @@ -25,8 +25,13 @@ public boolean matches(Object o) { } } else if (o instanceof Throwable) { if (o.toString().startsWith(prefix)) { + String input; Throwable throwable = (Throwable) o; - String input = throwable.getCause().toString(); + if (throwable.getCause() == null) { + input = o.toString(); + } else { + input = throwable.getCause().toString(); + } return pattern.matcher(input).matches(); } } diff --git a/flexmark-util/src/test/java/com/vladsch/flexmark/util/options/AttributesTest.java b/flexmark-util/src/test/java/com/vladsch/flexmark/util/options/AttributesTest.java index bfe5c9a5f..79bb38f2a 100644 --- a/flexmark-util/src/test/java/com/vladsch/flexmark/util/options/AttributesTest.java +++ b/flexmark-util/src/test/java/com/vladsch/flexmark/util/options/AttributesTest.java @@ -1,6 +1,8 @@ package com.vladsch.flexmark.util.options; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttribute; +import com.vladsch.flexmark.util.html.MutableAttributes; import org.junit.Test; import static org.junit.Assert.assertEquals; @@ -8,7 +10,7 @@ public class AttributesTest { @Test public void testBasic() throws Exception { - Attributes attributes = new Attributes(); + MutableAttributes attributes = new MutableAttributes(); assertEquals("empty no attributes", false, attributes.contains("class")); assertEquals("empty no values", false, attributes.containsValue("class", "class1")); diff --git a/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/LineAppendableImplTest.java b/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/LineAppendableImplTest.java index 02b7beaa4..0360bb42d 100644 --- a/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/LineAppendableImplTest.java +++ b/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/LineAppendableImplTest.java @@ -1,8 +1,11 @@ package com.vladsch.flexmark.util.sequence; +import com.vladsch.flexmark.util.ExceptionMatcher; import com.vladsch.flexmark.util.sequence.builder.SequenceBuilder; import com.vladsch.flexmark.util.sequence.mappers.SpaceMapper; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.ExpectedException; import java.util.ArrayList; @@ -10,6 +13,9 @@ @SuppressWarnings("PointlessArithmeticExpression") public class LineAppendableImplTest { + @Rule + public ExpectedException thrown = ExpectedException.none(); + @Test public void test_emptyAppendableIterator() { @@ -2021,6 +2027,55 @@ public void test_appendLineAppendablePrefixedUnterminated() { "", fa2.toString()); } + @Test + public void test_appendLineAppendablePrefixedUnterminated2() { + String input = "" + + "0:2343568\n" + + "1:2343568\n" + + "2:2343568\n" + + "3:2343568\n" + + "4:2343568\n" + + ""; + BasedSequence sequence = BasedSequence.of(input); + LineAppendableImpl fa = new LineAppendableImpl(SequenceBuilder.emptyBuilder(sequence), 0); + + fa.append("> "); + assertEquals("" + + "> " + + "", fa.toString()); + + fa.setPrefixLength(0, 2); + assertEquals("" + + "> \n" + + "", fa.toString()); + + LineInfo info = fa.getLineInfo(0); + assertEquals(3, info.length); + assertEquals(2, info.prefixLength); + assertEquals(0, info.textLength); + } + + @Test + public void test_appendLineAppendablePrefixedUnterminated3() { + String input = "" + + "0:2343568\n" + + "1:2343568\n" + + "2:2343568\n" + + "3:2343568\n" + + "4:2343568\n" + + ""; + BasedSequence sequence = BasedSequence.of(input); + LineAppendableImpl fa = new LineAppendableImpl(SequenceBuilder.emptyBuilder(sequence), 0); + + fa.append("> "); + assertEquals("" + + "> " + + "", fa.toString()); + + thrown.expect(ExceptionMatcher.match(IllegalArgumentException.class, "prefixLength 3 is out of valid range [0, 3) for the line")); + fa.setPrefixLength(0, 3); + } + @Test public void test_appendLineAppendablePrefixedNoPrefixes() { String input = "" + diff --git a/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/PlaceholderReplacerTest.java b/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/PlaceholderReplacerTest.java index 058891b30..d717ea853 100644 --- a/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/PlaceholderReplacerTest.java +++ b/flexmark-util/src/test/java/com/vladsch/flexmark/util/sequence/PlaceholderReplacerTest.java @@ -8,7 +8,7 @@ import java.util.function.BiConsumer; import java.util.function.Function; -import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertArrayEquals; public class PlaceholderReplacerTest { static List spansOf(String... spans) { @@ -41,7 +41,7 @@ public void test_simple() { List params = spansOf("", ""); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "Joe Smith", "" }, spansOf(params)); + assertArrayEquals(new String[] { "Joe Smith", "" }, spansOf(params)); } @Test @@ -53,7 +53,7 @@ public void test_spanSimple2() { List params = spansOf(""); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "", "Joe Smith" }, spansOf(params)); + assertArrayEquals(new String[] { "", "Joe Smith" }, spansOf(params)); } @Test @@ -65,7 +65,7 @@ public void test_spanSimple3() { List params = spansOf("<", "USER", ">"); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "", "", "" }, spansOf(params)); + assertArrayEquals(new String[] { "", "", "" }, spansOf(params)); } @Test @@ -77,7 +77,7 @@ public void test_spanComplex() { List params = spansOf(" <", "USER", ">"); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); + assertArrayEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); } @Test @@ -89,7 +89,7 @@ public void test_spanComplex2() { List params = spansOf(" "); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); + assertArrayEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); } @Test @@ -101,6 +101,6 @@ public void test_spanUndefined() { List params = spansOf(" "); PlaceholderReplacer.replaceAll(params, map::get, '<', '>', ourGetter, ourSetter); - assertEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); + assertArrayEquals(new String[] { "", "Joe Smith ", "", "" }, spansOf(params)); } } diff --git a/flexmark-youtrack-converter/flexmark-youtrack-converter.iml b/flexmark-youtrack-converter/flexmark-youtrack-converter.iml index 8dcc9cbf7..8f307c1e6 100644 --- a/flexmark-youtrack-converter/flexmark-youtrack-converter.iml +++ b/flexmark-youtrack-converter/flexmark-youtrack-converter.iml @@ -21,9 +21,9 @@ - + - + \ No newline at end of file diff --git a/flexmark-youtrack-converter/pom.xml b/flexmark-youtrack-converter/pom.xml index 90585826a..6fb9aae86 100644 --- a/flexmark-youtrack-converter/pom.xml +++ b/flexmark-youtrack-converter/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-youtrack-converter diff --git a/flexmark/flexmark.iml b/flexmark/flexmark.iml index 2df7dd377..208458800 100644 --- a/flexmark/flexmark.iml +++ b/flexmark/flexmark.iml @@ -9,16 +9,11 @@ - - - - - - + @@ -34,7 +29,9 @@ - + + + - + \ No newline at end of file diff --git a/flexmark/pom.xml b/flexmark/pom.xml index a6db4e825..8bebffb4d 100644 --- a/flexmark/pom.xml +++ b/flexmark/pom.xml @@ -4,7 +4,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark @@ -73,9 +73,21 @@ test - log4j - log4j - 1.2.17 + org.apache.logging.log4j + log4j-api + 2.20.0 + test + + + org.apache.logging.log4j + log4j-core + 2.20.0 + test + + + org.apache.logging.log4j + log4j-1.2-api + 2.20.0 test diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/Code.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/Code.java index c88b6854f..8f367c6aa 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/Code.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/Code.java @@ -1,6 +1,7 @@ package com.vladsch.flexmark.ast; import com.vladsch.flexmark.util.ast.DoNotLinkDecorate; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.builder.ISequenceBuilder; import org.jetbrains.annotations.NotNull; @@ -23,7 +24,7 @@ public void getAstExtra(@NotNull StringBuilder out) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { out.append(getText()); return false; } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/FencedCodeBlock.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/FencedCodeBlock.java index 5d7daa0bc..a58e02b2a 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/FencedCodeBlock.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/FencedCodeBlock.java @@ -90,11 +90,11 @@ public void setAttributes(BasedSequence attributes) { public BasedSequence getInfoDelimitedByAny(CharPredicate delimiters) { BasedSequence language = BasedSequence.NULL; if (info.isNotNull() && !info.isBlank()) { - int space = info.indexOfAny(delimiters); - if (space == -1) { + int delimiter = info.indexOfAny(delimiters); + if (delimiter == -1) { language = info; } else { - language = info.subSequence(0, space); + language = info.subSequence(0, delimiter); } } return language; diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/HardLineBreak.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/HardLineBreak.java index 9d410fc94..b3d4dfa7b 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/HardLineBreak.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/HardLineBreak.java @@ -2,6 +2,7 @@ import com.vladsch.flexmark.util.ast.DoNotTrim; import com.vladsch.flexmark.util.ast.Node; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.builder.ISequenceBuilder; @@ -22,7 +23,7 @@ public HardLineBreak(BasedSequence chars) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { BasedSequence chars = getChars(); out.add(chars.subSequence(chars.length() - 1, chars.length())); return false; diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/Heading.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/Heading.java index dc97b5e4f..fcec361ba 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/Heading.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/Heading.java @@ -1,8 +1,10 @@ package com.vladsch.flexmark.ast; +import com.vladsch.flexmark.html.HtmlRenderer; import com.vladsch.flexmark.util.ast.Block; import com.vladsch.flexmark.util.ast.BlockContent; import com.vladsch.flexmark.util.ast.TextCollectingVisitor; +import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import org.jetbrains.annotations.NotNull; @@ -29,7 +31,10 @@ public BasedSequence[] getSegments() { @Override public String getAnchorRefText() { - return new TextCollectingVisitor().collectAndGetText(this).trim(); + boolean trimLeadingSpaces = HtmlRenderer.HEADER_ID_REF_TEXT_TRIM_LEADING_SPACES.get(getDocument()); + boolean trimTrailingSpaces = HtmlRenderer.HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES.get(getDocument()); + + return new TextCollectingVisitor().collectAndGetText(this, TextContainer.F_FOR_HEADING_ID + (trimLeadingSpaces ? 0 : TextContainer.F_NO_TRIM_REF_TEXT_START) + (trimTrailingSpaces ? 0 : TextContainer.F_NO_TRIM_REF_TEXT_END)); } @Override diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/HtmlEntity.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/HtmlEntity.java index 45b53d68f..11c3ed392 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/HtmlEntity.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/HtmlEntity.java @@ -1,6 +1,7 @@ package com.vladsch.flexmark.ast; import com.vladsch.flexmark.util.ast.Node; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Escaping; @@ -36,7 +37,7 @@ public HtmlEntity(BasedSequence chars) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { if (any(flags, F_NODE_TEXT)) { out.append(getChars()); } else { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/IndentedCodeBlock.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/IndentedCodeBlock.java index 2448acbab..53ae2c7a8 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/IndentedCodeBlock.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/IndentedCodeBlock.java @@ -2,12 +2,19 @@ import com.vladsch.flexmark.util.ast.Block; import com.vladsch.flexmark.util.ast.BlockContent; +import com.vladsch.flexmark.util.ast.NodeVisitor; +import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; +import com.vladsch.flexmark.util.sequence.Escaping; +import com.vladsch.flexmark.util.sequence.ReplacedTextMapper; +import com.vladsch.flexmark.util.sequence.builder.ISequenceBuilder; import org.jetbrains.annotations.NotNull; import java.util.List; -public class IndentedCodeBlock extends Block { +import static com.vladsch.flexmark.util.misc.BitFieldSet.any; + +public class IndentedCodeBlock extends Block implements TextContainer { @NotNull @Override @@ -29,4 +36,19 @@ public IndentedCodeBlock(BasedSequence chars, List segments) { public IndentedCodeBlock(BlockContent blockContent) { super(blockContent); } + + @Override + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { + final BasedSequence chars = getContentChars(); + if (any(flags, F_NODE_TEXT)) { + out.append(chars); + } else { + ReplacedTextMapper textMapper = new ReplacedTextMapper(chars); + BasedSequence unescaped = Escaping.unescape(chars, textMapper); + if (!unescaped.isEmpty()) { + out.append(unescaped); + } + } + return false; + } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkNode.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkNode.java index 631fe83ed..09cd8e263 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkNode.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkNode.java @@ -1,6 +1,7 @@ package com.vladsch.flexmark.ast; import com.vladsch.flexmark.util.ast.DoNotLinkDecorate; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Escaping; @@ -17,7 +18,7 @@ public LinkNode(BasedSequence chars) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { int urlType = flags & F_LINK_TEXT_TYPE; BasedSequence url; diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRef.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRef.java index 02662db5c..4b9991d3a 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRef.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRef.java @@ -2,7 +2,7 @@ import com.vladsch.flexmark.util.sequence.BasedSequence; -public class LinkRef extends RefNode { +public class LinkRef extends RefNode implements LinkRendered { public LinkRef() { } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRefDerived.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRefDerived.java index f5fb69497..f57503453 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRefDerived.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRefDerived.java @@ -5,7 +5,7 @@ */ public interface LinkRefDerived { /** - * @return true if this node can be reverted to its text equivalent without loss of HTML rendering results + * @return true if this node will be rendered as text because it depends on a reference which is not defined. */ boolean isTentative(); } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRendered.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRendered.java new file mode 100644 index 000000000..b6f40b528 --- /dev/null +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/LinkRendered.java @@ -0,0 +1,8 @@ +package com.vladsch.flexmark.ast; + +/** + * Nodes which are textually derived from LinkRef + */ +public interface LinkRendered extends LinkRefDerived { + +} diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/Paragraph.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/Paragraph.java index a6f6b4e55..aa2ebdc37 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/Paragraph.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/Paragraph.java @@ -2,6 +2,7 @@ import com.vladsch.flexmark.util.ast.Block; import com.vladsch.flexmark.util.ast.BlockContent; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.builder.ISequenceBuilder; @@ -133,10 +134,14 @@ public boolean hasTableSeparator() { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { - if (!out.isEmpty()) { - out.add("\n\n"); - } + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { return true; } + + @Override + public void collectEndText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { + if (trailingBlankLine) { + out.add("\n"); + } + } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/RefNode.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/RefNode.java index 241fd7f8d..9c22a947d 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/RefNode.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/RefNode.java @@ -3,6 +3,7 @@ import com.vladsch.flexmark.ast.util.ReferenceRepository; import com.vladsch.flexmark.parser.Parser; import com.vladsch.flexmark.util.ast.*; +import com.vladsch.flexmark.util.misc.BitFieldSet; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Escaping; import com.vladsch.flexmark.util.sequence.ReplacedTextMapper; @@ -210,11 +211,24 @@ public void setReferenceClosingMarker(BasedSequence referenceClosingMarker) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { + // images no longer add alt text + int urlType = flags & F_LINK_TEXT_TYPE; if (urlType == F_LINK_TEXT) { - return true; + // To allow using leading/trailing spaces for generating heading ids, need to include stripped out spaces + if (BitFieldSet.any(flags, F_FOR_HEADING_ID) && this instanceof ImageRef) return false; + + if (BitFieldSet.any(flags, F_FOR_HEADING_ID) && BitFieldSet.any(flags, F_NO_TRIM_REF_TEXT_START | F_NO_TRIM_REF_TEXT_END)) { + BasedSequence[] segments = getSegments(); + if (BitFieldSet.any(flags, F_NO_TRIM_REF_TEXT_START)) out.append(getChars().baseSubSequence(segments[0].getEndOffset(), segments[1].getStartOffset())); + nodeVisitor.visitChildren(this); + if (BitFieldSet.any(flags, F_NO_TRIM_REF_TEXT_END)) out.append(getChars().baseSubSequence(segments[1].getEndOffset(), segments[2].getStartOffset())); + return false; + } else { + return true; + } } else { Reference reference = getReferenceNode(getDocument()); if (urlType == F_LINK_NODE_TEXT) { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/SoftLineBreak.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/SoftLineBreak.java index 079ae08c4..dcfab2013 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/SoftLineBreak.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/SoftLineBreak.java @@ -1,9 +1,6 @@ package com.vladsch.flexmark.ast; -import com.vladsch.flexmark.util.ast.DoNotAttributeDecorate; -import com.vladsch.flexmark.util.ast.DoNotTrim; -import com.vladsch.flexmark.util.ast.Node; -import com.vladsch.flexmark.util.ast.TextContainer; +import com.vladsch.flexmark.util.ast.*; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.builder.ISequenceBuilder; import org.jetbrains.annotations.NotNull; @@ -47,7 +44,7 @@ public void setCharsFromSegments() { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { out.add(getChars()); return false; } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/Text.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/Text.java index 583fa1bc6..fd9054dc1 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/Text.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/Text.java @@ -1,6 +1,7 @@ package com.vladsch.flexmark.ast; import com.vladsch.flexmark.util.ast.Node; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Escaping; @@ -34,13 +35,13 @@ public BasedSequence[] getSegments() { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { if (any(flags, F_NODE_TEXT)) { out.append(getChars()); } else { ReplacedTextMapper textMapper = new ReplacedTextMapper(getChars()); BasedSequence unescaped = Escaping.unescape(getChars(), textMapper); - out.append(unescaped); + if (!unescaped.isEmpty()) out.append(unescaped); } return false; } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/TextBase.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/TextBase.java index 7d802c2c5..99616ba98 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/TextBase.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/TextBase.java @@ -1,6 +1,7 @@ package com.vladsch.flexmark.ast; import com.vladsch.flexmark.util.ast.Node; +import com.vladsch.flexmark.util.ast.NodeVisitor; import com.vladsch.flexmark.util.ast.TextContainer; import com.vladsch.flexmark.util.sequence.BasedSequence; import com.vladsch.flexmark.util.sequence.Escaping; @@ -34,7 +35,7 @@ public void getAstExtra(@NotNull StringBuilder out) { } @Override - public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags) { + public boolean collectText(ISequenceBuilder, BasedSequence> out, int flags, NodeVisitor nodeVisitor) { if (any(flags, F_NODE_TEXT)) { out.append(getChars()); } else { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProviderAdapter.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProviderAdapter.java index d56a32569..ff6bc554e 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProviderAdapter.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProviderAdapter.java @@ -3,6 +3,7 @@ import com.vladsch.flexmark.html.renderer.AttributablePart; import com.vladsch.flexmark.util.ast.Node; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import com.vladsch.flexmark.util.visitor.AstActionHandler; import java.util.Collection; @@ -47,7 +48,7 @@ public AttributeProviderAdapter addHandler(AttributeProvidingHandler handler) { } @Override - public void setAttributes(Node node, AttributablePart part, Attributes attributes) { + public void setAttributes(Node node, AttributablePart part, MutableAttributes attributes) { processNode(node, false, (n, handler) -> handler.setAttributes(n, part, attributes)); } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProvidingHandler.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProvidingHandler.java index 05dcd7044..319a4b08c 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProvidingHandler.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/AttributeProvidingHandler.java @@ -2,7 +2,7 @@ import com.vladsch.flexmark.html.renderer.AttributablePart; import com.vladsch.flexmark.util.ast.Node; -import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import com.vladsch.flexmark.util.visitor.AstAction; import com.vladsch.flexmark.util.visitor.AstHandler; @@ -11,12 +11,12 @@ public AttributeProvidingHandler(Class aClass, AttributeProvidingVisitor a super(aClass, adapter); } - public void setAttributes(Node node, AttributablePart part, Attributes attributes) { + public void setAttributes(Node node, AttributablePart part, MutableAttributes attributes) { //noinspection unchecked getAdapter().setAttributes((N) node, part, attributes); } public static interface AttributeProvidingVisitor extends AstAction { - void setAttributes(N node, AttributablePart part, Attributes attributes); + void setAttributes(N node, AttributablePart part, MutableAttributes attributes); } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/Parsing.java b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/Parsing.java index 7e41b496e..c9478ee3a 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/ast/util/Parsing.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/ast/util/Parsing.java @@ -3,8 +3,14 @@ import com.vladsch.flexmark.parser.Parser; import com.vladsch.flexmark.util.data.DataHolder; import com.vladsch.flexmark.util.format.TableFormatOptions; +import com.vladsch.flexmark.util.misc.CharPredicate; import com.vladsch.flexmark.util.sequence.Escaping; +import com.vladsch.flexmark.util.sequence.SequenceUtils; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; +import java.util.HashMap; +import java.util.function.Function; import java.util.regex.Pattern; import static com.vladsch.flexmark.parser.Parser.*; @@ -13,248 +19,471 @@ public class Parsing { final public static char INTELLIJ_DUMMY_IDENTIFIER_CHAR = TableFormatOptions.INTELLIJ_DUMMY_IDENTIFIER_CHAR; final public static String INTELLIJ_DUMMY_IDENTIFIER = TableFormatOptions.INTELLIJ_DUMMY_IDENTIFIER; +// final public static String XML_NAMESPACE_START = "[_A-Za-z]"; +// final public static String XML_NAMESPACE_CHAR = XML_NAME_SPACE_START + "|-|.|[0-9]"; + final public static String XML_NAMESPACE_START = "[_A-Za-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]"; // excluded [#x10000-#xEFFFF] + final public static String XML_NAMESPACE_CHAR = XML_NAMESPACE_START + "|[.0-9\u00B7\u0300-\u036F\u203F-\u2040-]"; + final public static String XML_NAMESPACE = "(?:(?:" + XML_NAMESPACE_START + ")(?:" + XML_NAMESPACE_CHAR + ")*:)?"; + // save options for others to use when only parsing instance is available final public DataHolder options; + final private static String ST_EOL = "(?:\r\n|\r|\n)"; + final private static String ST_ESCAPED_CHAR = "\\\\" + Escaping.ESCAPABLE; + final private static Pattern ST_LINK_LABEL = Pattern.compile("^\\[(?:[^\\\\\\[\\]]|" + ST_ESCAPED_CHAR + "|\\\\){0,999}\\]"); + final private static String ST_LINK_TITLE_STRING = "(?:\"(" + ST_ESCAPED_CHAR + "|[^\"\\x00])*\"" + + '|' + + "'(" + ST_ESCAPED_CHAR + "|[^'\\x00])*'" + + '|' + + "\$(" + ST_ESCAPED_CHAR + "|[^)\\x00])*\$)"; + final private static Pattern ST_LINK_TITLE = Pattern.compile("^" + ST_LINK_TITLE_STRING); + final public String EOL = ST_EOL; + final public String ESCAPED_CHAR = ST_ESCAPED_CHAR; + final public Pattern LINK_LABEL = ST_LINK_LABEL; + final public Pattern LINK_DESTINATION_ANGLES; + final public String LINK_TITLE_STRING = ST_LINK_TITLE_STRING; + final public Pattern LINK_TITLE = ST_LINK_TITLE; + final public Pattern LINK_DESTINATION; + final public Pattern LINK_DESTINATION_MATCHED_PARENS; + final public Pattern LINK_DESTINATION_MATCHED_PARENS_NOSP; + + final private static String ST_EXCLUDED_0_TO_SPACE_IDI = "\u0000-\u001e\u0020"; + final private static String ST_EXCLUDED_0_TO_SPACE_NO_IDI = "\u0000-\u0020"; + final private static String ST_ADDITIONAL_CHARS_IDI = "\u001f"; + final private static String ST_ADDITIONAL_CHARS_NO_IDI = ""; + final private static String ST_ADDITIONAL_CHARS_SET_IDI = "[\u001f]"; + final private static String ST_ADDITIONAL_CHARS_SET_NO_IDI = ""; + + final public static String ST_HTMLCOMMENT = "|"; + final public static String ST_PROCESSINGINSTRUCTION = "[<][?].*?[?][>]"; + final public static String ST_CDATA = ""; + final public static String ST_SINGLEQUOTEDVALUE = "'[^']*'"; + final public static String ST_DOUBLEQUOTEDVALUE = "\"[^\"]*\""; + + final public String HTMLCOMMENT = ST_HTMLCOMMENT; + final public String PROCESSINGINSTRUCTION = ST_PROCESSINGINSTRUCTION; + final public String CDATA = ST_CDATA; + final public String SINGLEQUOTEDVALUE = ST_SINGLEQUOTEDVALUE; + final public String DOUBLEQUOTEDVALUE = ST_DOUBLEQUOTEDVALUE; + + final private static String ST_ASCII_PUNCTUATION = "'!\"#\\$%&\\*\\+,\\-\\./:;=\\?@\\\\\\^_`\\|~"; + final private static String ST_ASCII_OPEN_PUNCTUATION = "\$<\\[\\{"; + final private static String ST_ASCII_CLOSE_PUNCTUATION = "\$>\\]\\}"; + final private static Pattern ST_PUNCTUATION = Pattern.compile( + "^[" + ST_ASCII_PUNCTUATION + ST_ASCII_OPEN_PUNCTUATION + ST_ASCII_CLOSE_PUNCTUATION + "\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]"); + final private static Pattern ST_PUNCTUATION_OPEN = Pattern.compile( + "^[" + ST_ASCII_PUNCTUATION + ST_ASCII_OPEN_PUNCTUATION + "]|[\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ST_ASCII_CLOSE_PUNCTUATION + "]"); + final private static Pattern ST_PUNCTUATION_CLOSE = Pattern.compile( + "^[" + ST_ASCII_PUNCTUATION + ST_ASCII_CLOSE_PUNCTUATION + "]|[\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ST_ASCII_OPEN_PUNCTUATION + "]"); + final private static Pattern ST_PUNCTUATION_ONLY = Pattern.compile( + "^[" + ST_ASCII_PUNCTUATION + "\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ST_ASCII_OPEN_PUNCTUATION + ST_ASCII_CLOSE_PUNCTUATION + "]"); + final private static Pattern ST_PUNCTUATION_OPEN_ONLY = Pattern.compile( + "^[" + ST_ASCII_OPEN_PUNCTUATION + "]"); + final private static Pattern ST_PUNCTUATION_CLOSE_ONLY = Pattern.compile( + "^[" + ST_ASCII_CLOSE_PUNCTUATION + "]"); + + final public String ASCII_PUNCTUATION = ST_ASCII_PUNCTUATION; + final public String ASCII_OPEN_PUNCTUATION = ST_ASCII_OPEN_PUNCTUATION; + final public String ASCII_CLOSE_PUNCTUATION = ST_ASCII_CLOSE_PUNCTUATION; + + final public Pattern PUNCTUATION = ST_PUNCTUATION; + final public Pattern PUNCTUATION_OPEN = ST_PUNCTUATION_OPEN; + final public Pattern PUNCTUATION_CLOSE = ST_PUNCTUATION_CLOSE; + final public Pattern PUNCTUATION_ONLY = ST_PUNCTUATION_ONLY; + final public Pattern PUNCTUATION_OPEN_ONLY = ST_PUNCTUATION_OPEN_ONLY; + final public Pattern PUNCTUATION_CLOSE_ONLY = ST_PUNCTUATION_CLOSE_ONLY; + + final private static Pattern ST_ESCAPABLE = Pattern.compile('^' + Escaping.ESCAPABLE); + final private static Pattern ST_TICKS = Pattern.compile("`+"); + final private static Pattern ST_TICKS_HERE = Pattern.compile("^`+"); + final private static Pattern ST_SPNL = Pattern.compile("^(?:[ \t])*(?:" + ST_EOL + "(?:[ \t])*)?"); + final private static Pattern ST_SPNL_URL = Pattern.compile("^(?:[ \t])*" + ST_EOL); + final private static Pattern ST_SPNI = Pattern.compile("^ {0,3}"); + final private static Pattern ST_SP = Pattern.compile("^(?:[ \t])*"); + final private static Pattern ST_REST_OF_LINE = Pattern.compile("^.*" + ST_EOL); + final private static Pattern ST_UNICODE_WHITESPACE_CHAR = Pattern.compile("^[\\p{Zs}\t\r\n\f]"); + final private static Pattern ST_WHITESPACE = Pattern.compile("\\s+"); + final private static Pattern ST_FINAL_SPACE = Pattern.compile(" *$"); + final private static Pattern ST_LINE_END = Pattern.compile("^[ \t]*(?:" + ST_EOL + "|$)"); + final private static Pattern ST_LINK_DESTINATION_ANGLES_SPC = Pattern.compile("^(?:[<](?:[^<> \\t\\n\\\\\\x00]" + '|' + ST_ESCAPED_CHAR + '|' + "\\\\| (?![\"']))*[>])"); + final private static Pattern ST_LINK_DESTINATION_ANGLES_NO_SPC = Pattern.compile("^(?:[<](?:[^<> \\t\\n\\\\\\x00]" + '|' + ST_ESCAPED_CHAR + '|' + "\\\\)*[>])"); + + final public Pattern ESCAPABLE = ST_ESCAPABLE; + final public Pattern TICKS = ST_TICKS; + final public Pattern TICKS_HERE = ST_TICKS_HERE; + final public Pattern EMAIL_AUTOLINK; + final public Pattern AUTOLINK; + final public Pattern WWW_AUTOLINK; + final public Pattern SPNL = ST_SPNL; + final public Pattern SPNL_URL = ST_SPNL_URL; + final public Pattern SPNI = ST_SPNI; + final public Pattern SP = ST_SP; + final public Pattern REST_OF_LINE = ST_REST_OF_LINE; + final public Pattern UNICODE_WHITESPACE_CHAR = ST_UNICODE_WHITESPACE_CHAR; + final public Pattern WHITESPACE = ST_WHITESPACE; + final public Pattern FINAL_SPACE = ST_FINAL_SPACE; + final public Pattern LINE_END = ST_LINE_END; + + // IntelliJDummyIdentifier dependent + final private static String ST_TAGNAME_IDI = "[A-Za-z" + ST_ADDITIONAL_CHARS_IDI + "][A-Za-z0-9" + ST_ADDITIONAL_CHARS_IDI + "-]*"; + final private static String ST_TAGNAME_NO_IDI = "[A-Za-z" + ST_ADDITIONAL_CHARS_NO_IDI + "][A-Za-z0-9" + ST_ADDITIONAL_CHARS_NO_IDI + "-]*"; + + final private static String ST_UNQUOTEDVALUE_IDI = "[^\"'=<>{}`" + ST_EXCLUDED_0_TO_SPACE_IDI + "]+"; + final private static String ST_UNQUOTEDVALUE_NO_IDI = "[^\"'=<>{}`" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]+"; + + final private static String ST_ATTRIBUTENAME_IDI = "[a-zA-Z" + ST_ADDITIONAL_CHARS_IDI + "_:][a-zA-Z0-9" + ST_ADDITIONAL_CHARS_IDI + ":._-]*"; + final private static String ST_ATTRIBUTENAME_NO_IDI = "[a-zA-Z" + ST_ADDITIONAL_CHARS_NO_IDI + "_:][a-zA-Z0-9" + ST_ADDITIONAL_CHARS_NO_IDI + ":._-]*"; + + final private static String ST_ATTRIBUTEVALUE_IDI = "(?:" + ST_UNQUOTEDVALUE_IDI + "|" + ST_SINGLEQUOTEDVALUE + "|" + ST_DOUBLEQUOTEDVALUE + ")"; + final private static String ST_ATTRIBUTEVALUE_NO_IDI = "(?:" + ST_UNQUOTEDVALUE_NO_IDI + "|" + ST_SINGLEQUOTEDVALUE + "|" + ST_DOUBLEQUOTEDVALUE + ")"; + + final private static String ST_ATTRIBUTEVALUESPEC_IDI = "(?:" + "\\s*=" + "\\s*" + ST_ATTRIBUTEVALUE_IDI + ")"; + final private static String ST_ATTRIBUTEVALUESPEC_NO_IDI = "(?:" + "\\s*=" + "\\s*" + ST_ATTRIBUTEVALUE_NO_IDI + ")"; + + final private static String ST_CLOSETAG_IDI = "]"; + final private static String ST_CLOSETAG_NO_IDI = "]"; + final private static String ST_NS_CLOSETAG_IDI = "]"; + final private static String ST_NS_CLOSETAG_NO_IDI = "]"; + + final private static String ST_ATTRIBUTE_IDI = "(?:" + "\\s+" + ST_ATTRIBUTENAME_IDI + ST_ATTRIBUTEVALUESPEC_IDI + "?)"; + final private static String ST_ATTRIBUTE_NO_IDI = "(?:" + "\\s+" + ST_ATTRIBUTENAME_NO_IDI + ST_ATTRIBUTEVALUESPEC_NO_IDI + "?)"; + + final private static String ST_DECLARATION_IDI = "]*>"; + final private static String ST_DECLARATION_NO_IDI = "]*>"; + + final private static String ST_ENTITY_IDI = "&(?:#x[a-f0-9" + ST_ADDITIONAL_CHARS_IDI + "]{1,8}|#[0-9]{1,8}|[a-z" + ST_ADDITIONAL_CHARS_IDI + "][a-z0-9" + ST_ADDITIONAL_CHARS_IDI + "]{1,31});"; + final private static String ST_ENTITY_NO_IDI = "&(?:#x[a-f0-9" + ST_ADDITIONAL_CHARS_NO_IDI + "]{1,8}|#[0-9]{1,8}|[a-z" + ST_ADDITIONAL_CHARS_NO_IDI + "][a-z0-9" + ST_ADDITIONAL_CHARS_NO_IDI + "]{1,31});"; + + final private static String ST_IN_BRACES_W_SP_IDI = "\\{\\{(?:[^{}\\\\" + ST_EXCLUDED_0_TO_SPACE_IDI + "]| |\t)*\\}\\}"; + final private static String ST_IN_BRACES_W_SP_NO_IDI = "\\{\\{(?:[^{}\\\\" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]| |\t)*\\}\\}"; + + final private static String ST_REG_CHAR_IDI = "[^\\\$)" + ST_EXCLUDED_0_TO_SPACE_IDI + "]"; + final private static String ST_REG_CHAR_NO_IDI = "[^\\\\()" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]"; + + final private static String ST_IN_MATCHED_PARENS_NOSP_IDI = "\\((" + ST_REG_CHAR_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + final private static String ST_IN_MATCHED_PARENS_NOSP_NO_IDI = "\$(" + ST_REG_CHAR_NO_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + + final private static String ST_REG_CHAR_SP_IDI = "[^\\\$)" + ST_EXCLUDED_0_TO_SPACE_IDI + "]| (?![\"'])"; + final private static String ST_REG_CHAR_SP_NO_IDI = "[^\\\\()" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]| (?![\"'])"; + + final private static String ST_IN_MATCHED_PARENS_W_SP_IDI = "\\((" + ST_REG_CHAR_SP_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + final private static String ST_IN_MATCHED_PARENS_W_SP_NO_IDI = "\$(" + ST_REG_CHAR_SP_NO_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + + final private static String ST_IN_PARENS_NOSP_IDI = "\$(" + ST_REG_CHAR_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + final private static String ST_IN_PARENS_NOSP_NO_IDI = "\$(" + ST_REG_CHAR_NO_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + + final private static String ST_IN_PARENS_W_SP_IDI = "\$(" + ST_REG_CHAR_SP_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + final private static String ST_IN_PARENS_W_SP_NO_IDI = "\$(" + ST_REG_CHAR_SP_NO_IDI + '|' + ST_ESCAPED_CHAR + ")*\$"; + + final private static String ST_OPENTAG_IDI = "<" + ST_TAGNAME_IDI + ST_ATTRIBUTE_IDI + "*" + "\\s*/?>"; + final private static String ST_OPENTAG_NO_IDI = "<" + ST_TAGNAME_NO_IDI + ST_ATTRIBUTE_NO_IDI + "*" + "\\s*/?>"; + final private static String ST_NS_OPENTAG_IDI = "<" + XML_NAMESPACE + ST_TAGNAME_IDI + ST_ATTRIBUTE_IDI + "*" + "\\s*/?>"; + final private static String ST_NS_OPENTAG_NO_IDI = "<" + XML_NAMESPACE + ST_TAGNAME_NO_IDI + ST_ATTRIBUTE_NO_IDI + "*" + "\\s*/?>"; + + final private static String ST_REG_CHAR_PARENS_IDI = "[^\\\\" + ST_EXCLUDED_0_TO_SPACE_IDI + "]"; + final private static String ST_REG_CHAR_PARENS_NO_IDI = "[^\\\\" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]"; + + final private static String ST_REG_CHAR_SP_PARENS_IDI = "[^\\\\" + ST_EXCLUDED_0_TO_SPACE_IDI + "]| (?![\"'])"; + final private static String ST_REG_CHAR_SP_PARENS_NO_IDI = "[^\\\\" + ST_EXCLUDED_0_TO_SPACE_NO_IDI + "]| (?![\"'])"; + + final private static Pattern ST_ENTITY_HERE_IDI = Pattern.compile('^' + ST_ENTITY_IDI, Pattern.CASE_INSENSITIVE); + final private static Pattern ST_ENTITY_HERE_NO_IDI = Pattern.compile('^' + ST_ENTITY_NO_IDI, Pattern.CASE_INSENSITIVE); + final public String ADDITIONAL_CHARS; final public String EXCLUDED_0_TO_SPACE; - - final public String EOL; - final public String ESCAPED_CHAR; - final public Pattern LINK_LABEL; - final public Pattern LINK_DESTINATION_ANGLES; - final public String LINK_TITLE_STRING; - final public Pattern LINK_TITLE; final public String REG_CHAR; + final public String REG_CHAR_PARENS; final public String REG_CHAR_SP; + final public String REG_CHAR_SP_PARENS; final public String IN_PARENS_NOSP; final public String IN_PARENS_W_SP; + final public String IN_MATCHED_PARENS_NOSP; + final public String IN_MATCHED_PARENS_W_SP; final public String IN_BRACES_W_SP; - final public Pattern LINK_DESTINATION; - final public Pattern LINK_DESTINATION_MATCHED_PARENS; - final public Pattern LINK_DESTINATION_MATCHED_PARENS_NOSP; - final public String HTMLCOMMENT; - final public String PROCESSINGINSTRUCTION; final public String DECLARATION; - final public String CDATA; final public String ENTITY; - - final public Pattern ENTITY_HERE; - final public String ASCII_PUNCTUATION; - final public String ASCII_OPEN_PUNCTUATION; - final public String ASCII_CLOSE_PUNCTUATION; - final public Pattern PUNCTUATION; - final public Pattern PUNCTUATION_OPEN; - final public Pattern PUNCTUATION_CLOSE; - final public Pattern PUNCTUATION_ONLY; - final public Pattern PUNCTUATION_OPEN_ONLY; - final public Pattern PUNCTUATION_CLOSE_ONLY; - - //final public Pattern HTML_COMMENT; - final public Pattern ESCAPABLE; - final public Pattern TICKS; - final public Pattern TICKS_HERE; - final public Pattern EMAIL_AUTOLINK; - final public Pattern AUTOLINK; - final public Pattern WWW_AUTOLINK; - final public Pattern SPNL; - final public Pattern SPNL_URL; - final public Pattern SPNI; - final public Pattern SP; - final public Pattern REST_OF_LINE; - final public Pattern UNICODE_WHITESPACE_CHAR; - final public Pattern WHITESPACE; - final public Pattern FINAL_SPACE; - final public Pattern LINE_END; final public String TAGNAME; final public String ATTRIBUTENAME; final public String UNQUOTEDVALUE; - final public String SINGLEQUOTEDVALUE; - final public String DOUBLEQUOTEDVALUE; final public String ATTRIBUTEVALUE; final public String ATTRIBUTEVALUESPEC; final public String ATTRIBUTE; - final public String OPENTAG; final public String CLOSETAG; + final public String HTMLTAG; - final public Pattern HTML_TAG; + final public Pattern ENTITY_HERE; + final public Pattern HTML_TAG; final public Pattern LIST_ITEM_MARKER; final public int CODE_BLOCK_INDENT; + final public boolean intellijDummyIdentifier; final public boolean htmlForTranslator; final public String translationHtmlInlineTagPattern; final public String translationAutolinkTagPattern; - final public String INVALID_LINK_CHARS; - final public String IN_MATCHED_PARENS_NOSP; - final public String IN_MATCHED_PARENS_W_SP; - final public String REG_CHAR_PARENS; - final public String REG_CHAR_SP_PARENS; + final public boolean spaceInLinkUrl; + final public boolean parseJekyllMacroInLinkUrl; + final public String itemPrefixChars; + final public boolean listsItemMarkerSpace; + final public boolean listsOrderedItemDotOnly; + final public boolean allowNameSpace; + + private static class PatternTypeFlags { + final @Nullable Boolean intellijDummyIdentifier; + final @Nullable Boolean htmlForTranslator; + final @Nullable String translationHtmlInlineTagPattern; + final @Nullable String translationAutolinkTagPattern; + final @Nullable Boolean spaceInLinkUrl; + final @Nullable Boolean parseJekyllMacroInLinkUrl; + final @Nullable String itemPrefixChars; + final @Nullable Boolean listsItemMarkerSpace; + final @Nullable Boolean listsOrderedItemDotOnly; + final @Nullable Boolean allowNameSpace; + + PatternTypeFlags(DataHolder options) { + this.intellijDummyIdentifier = Parser.INTELLIJ_DUMMY_IDENTIFIER.get(options); + this.htmlForTranslator = Parser.HTML_FOR_TRANSLATOR.get(options); + this.translationHtmlInlineTagPattern = Parser.TRANSLATION_HTML_INLINE_TAG_PATTERN.get(options); + this.translationAutolinkTagPattern = Parser.TRANSLATION_AUTOLINK_TAG_PATTERN.get(options); + this.spaceInLinkUrl = Parser.SPACE_IN_LINK_URLS.get(options); + this.parseJekyllMacroInLinkUrl = Parser.PARSE_JEKYLL_MACROS_IN_URLS.get(options); + this.itemPrefixChars = LISTS_ITEM_PREFIX_CHARS.get(options); + this.listsItemMarkerSpace = LISTS_ITEM_MARKER_SPACE.get(options); + this.listsOrderedItemDotOnly = LISTS_ORDERED_ITEM_DOT_ONLY.get(options); + this.allowNameSpace = HTML_ALLOW_NAME_SPACE.get(options); + } + + public PatternTypeFlags( + @Nullable Boolean intellijDummyIdentifier + , @Nullable Boolean htmlForTranslator + , @Nullable String translationHtmlInlineTagPattern + , @Nullable String translationAutolinkTagPattern + , @Nullable Boolean spaceInLinkUrl + , @Nullable Boolean parseJekyllMacroInLinkUrl + , @Nullable String itemPrefixChars + , @Nullable Boolean listsItemMarkerSpace + , @Nullable Boolean listsOrderedItemDotOnly + , @Nullable Boolean allowNameSpace + ) { + this.intellijDummyIdentifier = intellijDummyIdentifier; + this.htmlForTranslator = htmlForTranslator; + this.translationHtmlInlineTagPattern = translationHtmlInlineTagPattern; + this.translationAutolinkTagPattern = translationAutolinkTagPattern; + this.spaceInLinkUrl = spaceInLinkUrl; + this.parseJekyllMacroInLinkUrl = parseJekyllMacroInLinkUrl; + this.itemPrefixChars = itemPrefixChars; + this.listsItemMarkerSpace = listsItemMarkerSpace; + this.listsOrderedItemDotOnly = listsOrderedItemDotOnly; + this.allowNameSpace = allowNameSpace; + } + + PatternTypeFlags withJekyllMacroInLinkUrl() { return new PatternTypeFlags(intellijDummyIdentifier, null, null, null, null, parseJekyllMacroInLinkUrl, null, null, null, null); } + + PatternTypeFlags withJekyllMacroSpaceInLinkUrl() { return new PatternTypeFlags(intellijDummyIdentifier, null, null, null, spaceInLinkUrl, parseJekyllMacroInLinkUrl, null, null, null, null); } + + PatternTypeFlags withHtmlTranslator() { return new PatternTypeFlags(intellijDummyIdentifier, htmlForTranslator, translationHtmlInlineTagPattern, translationAutolinkTagPattern, null, null, null, null, null, null); } + + PatternTypeFlags withItemPrefixChars() { return new PatternTypeFlags(null, null, null, null, null, null, itemPrefixChars, listsItemMarkerSpace, listsOrderedItemDotOnly, null); } + + PatternTypeFlags withAllowNameSpace() { return new PatternTypeFlags(null, null, null, null, null, null, null, null, null, allowNameSpace); } + + /** + * Compare where null entry equals any other value + * + * @param o other + * @return true if equal + */ + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + + PatternTypeFlags that = (PatternTypeFlags) o; + + if (intellijDummyIdentifier != null && !intellijDummyIdentifier.equals(that.intellijDummyIdentifier)) return false; + if (htmlForTranslator != null && !htmlForTranslator.equals(that.htmlForTranslator)) return false; + if (translationHtmlInlineTagPattern != null && !translationHtmlInlineTagPattern.equals(that.translationHtmlInlineTagPattern)) return false; + if (translationAutolinkTagPattern != null && !translationAutolinkTagPattern.equals(that.translationAutolinkTagPattern)) return false; + if (spaceInLinkUrl != null && !spaceInLinkUrl.equals(that.spaceInLinkUrl)) return false; + if (parseJekyllMacroInLinkUrl != null && !parseJekyllMacroInLinkUrl.equals(that.parseJekyllMacroInLinkUrl)) return false; + if (itemPrefixChars != null && !itemPrefixChars.equals(that.itemPrefixChars)) return false; + if (listsItemMarkerSpace != null && !listsItemMarkerSpace.equals(that.listsItemMarkerSpace)) return false; + if (allowNameSpace != null && !allowNameSpace.equals(that.allowNameSpace)) return false; + + return listsOrderedItemDotOnly == null || listsOrderedItemDotOnly.equals(that.listsOrderedItemDotOnly); + } + + @Override + public int hashCode() { + int result = intellijDummyIdentifier != null ? intellijDummyIdentifier.hashCode() : 0; + result = 31 * result + (htmlForTranslator != null ? htmlForTranslator.hashCode() : 0); + result = 31 * result + (translationHtmlInlineTagPattern != null ? translationHtmlInlineTagPattern.hashCode() : 0); + result = 31 * result + (translationAutolinkTagPattern != null ? translationAutolinkTagPattern.hashCode() : 0); + result = 31 * result + (spaceInLinkUrl != null ? spaceInLinkUrl.hashCode() : 0); + result = 31 * result + (parseJekyllMacroInLinkUrl != null ? parseJekyllMacroInLinkUrl.hashCode() : 0); + result = 31 * result + (itemPrefixChars != null ? itemPrefixChars.hashCode() : 0); + result = 31 * result + (listsItemMarkerSpace != null ? listsItemMarkerSpace.hashCode() : 0); + result = 31 * result + (listsOrderedItemDotOnly != null ? listsOrderedItemDotOnly.hashCode() : 0); + result = 31 * result + (allowNameSpace != null ? allowNameSpace.hashCode() : 0); + return result; + } + } + + final static HashMap> cachedPatterns = new HashMap<>(); + + static Pattern getCachedPattern(@NotNull String patternName, @NotNull PatternTypeFlags cachedTypeFlags, @NotNull Function factory) { + HashMap patternMap = cachedPatterns.computeIfAbsent(patternName, (key) -> new HashMap<>()); + return patternMap.computeIfAbsent(cachedTypeFlags, factory); + } public Parsing(DataHolder options) { this.options = options; - this.intellijDummyIdentifier = Parser.INTELLIJ_DUMMY_IDENTIFIER.get(options); - this.htmlForTranslator = Parser.HTML_FOR_TRANSLATOR.get(options); - this.translationHtmlInlineTagPattern = Parser.TRANSLATION_HTML_INLINE_TAG_PATTERN.get(options); - this.translationAutolinkTagPattern = Parser.TRANSLATION_AUTOLINK_TAG_PATTERN.get(options); - - this.EOL = "(?:\r\n|\r|\n)"; - this.ADDITIONAL_CHARS = ADDITIONAL_CHARS(); - this.EXCLUDED_0_TO_SPACE = EXCLUDED_0_TO_SPACE(); - - this.ESCAPED_CHAR = "\\\\" + Escaping.ESCAPABLE; - this.LINK_LABEL = Pattern - .compile("^\\[(?:[^\\\\\\[\\]]|" + ESCAPED_CHAR + "|\\\\){0,999}\\]"); - - this.LINK_DESTINATION_ANGLES = Parser.SPACE_IN_LINK_URLS.get(options) - ? Pattern.compile("^(?:[<](?:[^<> \\t\\n\\\\\\x00]" + '|' + ESCAPED_CHAR + '|' + "\\\\| (?![\"']))*[>])") - : Pattern.compile("^(?:[<](?:[^<> \\t\\n\\\\\\x00]" + '|' + ESCAPED_CHAR + '|' + "\\\\)*[>])"); - - //this.LINK_DESTINATION_ANGLES = Parser.SPACE_IN_LINK_URLS.getFrom(options) - // ? Pattern.compile("^(?:[<](?:" + - // "(?:[^<> \\t\\n\\\\\\x00]" + '|' + ESCAPED_CHAR + '|' + "\\\\| (?![\"]))" + - // (htmlForTranslator ? "|(?:_\\d+_)" : "") + - // ")*[>])") - // : Pattern.compile("^(?:[<](?:" + - // "(?:[^<> \\t\\n\\\\\\x00]" + '|' + ESCAPED_CHAR + '|' + "\\\\)" + - // (htmlForTranslator ? "|(?:_\\d+_)" : "") + - // ")*[>])"); - - this.LINK_TITLE_STRING = "(?:\"(" + ESCAPED_CHAR + "|[^\"\\x00])*\"" + - '|' + - "'(" + ESCAPED_CHAR + "|[^'\\x00])*'" + - '|' + - "\$(" + ESCAPED_CHAR + "|[^)\\x00])*\$)"; - this.LINK_TITLE = Pattern.compile("^" + LINK_TITLE_STRING); - this.REG_CHAR = "[^\\\$)" + EXCLUDED_0_TO_SPACE + "]"; - this.REG_CHAR_PARENS = "[^\\\\" + EXCLUDED_0_TO_SPACE + "]"; - this.REG_CHAR_SP = "[^\\\\()" + EXCLUDED_0_TO_SPACE + "]| (?![\"'])"; - this.REG_CHAR_SP_PARENS = "[^\\\\" + EXCLUDED_0_TO_SPACE + "]| (?![\"'])"; - this.IN_PARENS_NOSP = "\\((" + REG_CHAR + '|' + ESCAPED_CHAR + ")*\$"; - this.IN_PARENS_W_SP = "\$(" + REG_CHAR_SP + '|' + ESCAPED_CHAR + ")*\$"; - this.IN_MATCHED_PARENS_NOSP = "\$(" + REG_CHAR + '|' + ESCAPED_CHAR + ")*\$"; - this.IN_MATCHED_PARENS_W_SP = "\$(" + REG_CHAR_SP + '|' + ESCAPED_CHAR + ")*\$"; - this.IN_BRACES_W_SP = "\\{\\{(?:[^{}\\\\" + EXCLUDED_0_TO_SPACE + "]| |\t)*\\}\\}"; - this.LINK_DESTINATION = Pattern.compile( - "^(?:" + (Parser.PARSE_JEKYLL_MACROS_IN_URLS.get(options) ? IN_BRACES_W_SP + "|" : "") + - (Parser.SPACE_IN_LINK_URLS.get(options) ? "(?:" + REG_CHAR_SP + ")|" : REG_CHAR + "|") + - ESCAPED_CHAR + "|\\\\|" + (Parser.SPACE_IN_LINK_URLS.get(options) ? IN_PARENS_W_SP : IN_PARENS_NOSP) + ")*"); - this.LINK_DESTINATION_MATCHED_PARENS = Pattern.compile( - "^(?:" + (Parser.PARSE_JEKYLL_MACROS_IN_URLS.get(options) ? IN_BRACES_W_SP + "|" : "") - + (Parser.SPACE_IN_LINK_URLS.get(options) ? "(?:" + REG_CHAR_SP + ")|" : REG_CHAR + "|") + - ESCAPED_CHAR + "|\\\\|\$|\$)*"); - this.LINK_DESTINATION_MATCHED_PARENS_NOSP = Pattern.compile( - "^(?:" + (Parser.PARSE_JEKYLL_MACROS_IN_URLS.get(options) ? IN_BRACES_W_SP + "|" : "") - + (false ? "(?:" + REG_CHAR_SP + ")|" : REG_CHAR + "|") + - ESCAPED_CHAR + "|\\\\|\$|\$)*"); - this.HTMLCOMMENT = "|"; - this.PROCESSINGINSTRUCTION = "[<][?].*?[?][>]"; - this.DECLARATION = "]*>"; - this.CDATA = ""; - this.ENTITY = "&(?:#x[a-f0-9" + ADDITIONAL_CHARS + "]{1,8}|#[0-9]{1,8}|[a-z" + ADDITIONAL_CHARS + "][a-z0-9" + ADDITIONAL_CHARS + "]{1,31});"; - - this.ENTITY_HERE = Pattern.compile('^' + ENTITY, Pattern.CASE_INSENSITIVE); - this.ASCII_PUNCTUATION = "'!\"#\\$%&\\*\\+,\\-\\./:;=\\?@\\\\\\^_`\\|~"; - this.ASCII_OPEN_PUNCTUATION = "\$<\\[\\{"; - this.ASCII_CLOSE_PUNCTUATION = "\$>\\]\\}"; - this.PUNCTUATION = Pattern.compile( - "^[" + ASCII_PUNCTUATION + ASCII_OPEN_PUNCTUATION + ASCII_CLOSE_PUNCTUATION + "\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]"); - this.PUNCTUATION_OPEN = Pattern.compile( - "^[" + ASCII_PUNCTUATION + ASCII_OPEN_PUNCTUATION + "]|[\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ASCII_CLOSE_PUNCTUATION + "]"); - this.PUNCTUATION_CLOSE = Pattern.compile( - "^[" + ASCII_PUNCTUATION + ASCII_CLOSE_PUNCTUATION + "]|[\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ASCII_OPEN_PUNCTUATION + "]"); - this.PUNCTUATION_ONLY = Pattern.compile( - "^[" + ASCII_PUNCTUATION + "\\p{Pc}\\p{Pd}\\p{Pe}\\p{Pf}\\p{Pi}\\p{Po}\\p{Ps}]&&[^" + ASCII_OPEN_PUNCTUATION + ASCII_CLOSE_PUNCTUATION + "]"); - this.PUNCTUATION_OPEN_ONLY = Pattern.compile( - "^[" + ASCII_OPEN_PUNCTUATION + "]"); - this.PUNCTUATION_CLOSE_ONLY = Pattern.compile( - "^[" + ASCII_CLOSE_PUNCTUATION + "]"); - - //this.HTML_COMMENT = Pattern.compile(HTMLCOMMENT); - this.ESCAPABLE = Pattern.compile('^' + Escaping.ESCAPABLE); - this.TICKS = Pattern.compile("`+"); - this.TICKS_HERE = Pattern.compile("^`+"); - this.EMAIL_AUTOLINK = Pattern.compile( - "^<(" + - "(?:[a-zA-Z0-9" + ADDITIONAL_CHARS + ".!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9" + ADDITIONAL_CHARS + "](?:[a-zA-Z0-9" + ADDITIONAL_CHARS + "-]{0,61}[a-zA-Z0-9" + ADDITIONAL_CHARS + "])?(?:\\.[a-zA-Z0-9" + ADDITIONAL_CHARS + "](?:[a-zA-Z0-9" + ADDITIONAL_CHARS + "-]{0,61}[a-zA-Z0-9" + ADDITIONAL_CHARS + "])?)*)" + - (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + - ")>"); - - this.AUTOLINK = Pattern.compile( - "^<(" + - "(?:[a-zA-Z][a-zA-Z0-9" + ADDITIONAL_CHARS + ".+-]{1,31}:[^<>" + EXCLUDED_0_TO_SPACE + "]*)" + - (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + - ")>"); - - this.WWW_AUTOLINK = Pattern.compile( - "^<(" + - "(?:w" + ADDITIONAL_CHARS + "?){3,3}\\.[^<>" + EXCLUDED_0_TO_SPACE + "]*" + - (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + - ")>"); - - this.SPNL = Pattern.compile("^(?:[ \t])*(?:" + EOL + "(?:[ \t])*)?"); - this.SPNL_URL = Pattern.compile("^(?:[ \t])*" + EOL); - this.SPNI = Pattern.compile("^ {0,3}"); - this.SP = Pattern.compile("^(?:[ \t])*"); - this.REST_OF_LINE = Pattern.compile("^.*" + EOL); - this.UNICODE_WHITESPACE_CHAR = Pattern.compile("^[\\p{Zs}\t\r\n\f]"); - this.WHITESPACE = Pattern.compile("\\s+"); - this.FINAL_SPACE = Pattern.compile(" *$"); - this.LINE_END = Pattern.compile("^[ \t]*(?:" + EOL + "|$)"); - this.TAGNAME = "[A-Za-z" + ADDITIONAL_CHARS + "][A-Za-z0-9" + ADDITIONAL_CHARS + "-]*"; - this.ATTRIBUTENAME = "[a-zA-Z" + ADDITIONAL_CHARS + "_:][a-zA-Z0-9" + ADDITIONAL_CHARS + ":._-]*"; - this.UNQUOTEDVALUE = "[^\"'=<>{}`" + EXCLUDED_0_TO_SPACE + "]+"; - this.SINGLEQUOTEDVALUE = "'[^']*'"; - this.DOUBLEQUOTEDVALUE = "\"[^\"]*\""; - this.ATTRIBUTEVALUE = "(?:" + UNQUOTEDVALUE + "|" + SINGLEQUOTEDVALUE - + "|" + DOUBLEQUOTEDVALUE + ")"; - this.ATTRIBUTEVALUESPEC = "(?:" + "\\s*=" + "\\s*" + ATTRIBUTEVALUE + ")"; - this.ATTRIBUTE = "(?:" + "\\s+" + ATTRIBUTENAME + ATTRIBUTEVALUESPEC + "?)"; - - this.OPENTAG = "<" + TAGNAME + ATTRIBUTE + "*" + "\\s*/?>"; - this.CLOSETAG = "]"; - this.HTMLTAG = "(?:" + OPENTAG + "|" + CLOSETAG + "|" + HTMLCOMMENT - + "|" + PROCESSINGINSTRUCTION + "|" + DECLARATION + "|" + CDATA + - (htmlForTranslator ? "|<(?:" + translationHtmlInlineTagPattern + ")>|" : "") + ")"; - this.HTML_TAG = Pattern.compile('^' + HTMLTAG, Pattern.CASE_INSENSITIVE); - - final String itemPrefixChars = LISTS_ITEM_PREFIX_CHARS.get(options); - if (LISTS_ITEM_MARKER_SPACE.get(options)) { - if (LISTS_ORDERED_ITEM_DOT_ONLY.get(options)) { - this.LIST_ITEM_MARKER = Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?=[ \t])|^(\\d{1,9})([.])(?=[ \t])"); - } else { - this.LIST_ITEM_MARKER = Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?=[ \t])|^(\\d{1,9})([.)])(?=[ \t])"); - } + this.CODE_BLOCK_INDENT = Parser.CODE_BLOCK_INDENT.get(options); // make sure this is consistent with lists settings + PatternTypeFlags patternTypeFlags = new PatternTypeFlags(options); + this.intellijDummyIdentifier = Boolean.TRUE.equals(patternTypeFlags.intellijDummyIdentifier); + this.htmlForTranslator = Boolean.TRUE.equals(patternTypeFlags.htmlForTranslator); + this.translationHtmlInlineTagPattern = patternTypeFlags.translationHtmlInlineTagPattern; + this.translationAutolinkTagPattern = patternTypeFlags.translationAutolinkTagPattern; + this.spaceInLinkUrl = Boolean.TRUE.equals(patternTypeFlags.spaceInLinkUrl); + this.parseJekyllMacroInLinkUrl = Boolean.TRUE.equals(patternTypeFlags.parseJekyllMacroInLinkUrl); + this.itemPrefixChars = patternTypeFlags.itemPrefixChars; + this.listsItemMarkerSpace = Boolean.TRUE.equals(patternTypeFlags.listsItemMarkerSpace); + this.listsOrderedItemDotOnly = Boolean.TRUE.equals(patternTypeFlags.listsOrderedItemDotOnly); + this.allowNameSpace = Boolean.TRUE.equals(patternTypeFlags.allowNameSpace); + + if (intellijDummyIdentifier) { + this.ADDITIONAL_CHARS = ST_ADDITIONAL_CHARS_IDI; + this.EXCLUDED_0_TO_SPACE = ST_EXCLUDED_0_TO_SPACE_IDI; + this.REG_CHAR = ST_REG_CHAR_IDI; + this.REG_CHAR_PARENS = ST_REG_CHAR_PARENS_IDI; + this.REG_CHAR_SP = ST_REG_CHAR_SP_IDI; + this.REG_CHAR_SP_PARENS = ST_REG_CHAR_SP_PARENS_IDI; + this.IN_PARENS_NOSP = ST_IN_PARENS_NOSP_IDI; + this.IN_PARENS_W_SP = ST_IN_PARENS_W_SP_IDI; + this.IN_MATCHED_PARENS_NOSP = ST_IN_MATCHED_PARENS_NOSP_IDI; + this.IN_MATCHED_PARENS_W_SP = ST_IN_MATCHED_PARENS_W_SP_IDI; + this.IN_BRACES_W_SP = ST_IN_BRACES_W_SP_IDI; + this.DECLARATION = ST_DECLARATION_IDI; + this.ENTITY = ST_ENTITY_IDI; + this.TAGNAME = ST_TAGNAME_IDI; + this.ATTRIBUTENAME = ST_ATTRIBUTENAME_IDI; + this.UNQUOTEDVALUE = ST_UNQUOTEDVALUE_IDI; + this.ATTRIBUTEVALUE = ST_ATTRIBUTEVALUE_IDI; + this.ATTRIBUTEVALUESPEC = ST_ATTRIBUTEVALUESPEC_IDI; + this.ATTRIBUTE = ST_ATTRIBUTE_IDI; + this.OPENTAG = allowNameSpace ? ST_NS_OPENTAG_IDI : ST_OPENTAG_IDI; + this.CLOSETAG = allowNameSpace ? ST_NS_CLOSETAG_IDI : ST_CLOSETAG_IDI; } else { - if (LISTS_ORDERED_ITEM_DOT_ONLY.get(options)) { - this.LIST_ITEM_MARKER = Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?= |\t|$)|^(\\d{1,9})([.])(?= |\t|$)"); - } else { - this.LIST_ITEM_MARKER = Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?= |\t|$)|^(\\d{1,9})([.)])(?= |\t|$)"); - } + this.ADDITIONAL_CHARS = ST_ADDITIONAL_CHARS_NO_IDI; + this.EXCLUDED_0_TO_SPACE = ST_EXCLUDED_0_TO_SPACE_NO_IDI; + this.REG_CHAR = ST_REG_CHAR_NO_IDI; + this.REG_CHAR_PARENS = ST_REG_CHAR_PARENS_NO_IDI; + this.REG_CHAR_SP = ST_REG_CHAR_SP_NO_IDI; + this.REG_CHAR_SP_PARENS = ST_REG_CHAR_SP_PARENS_NO_IDI; + this.IN_PARENS_NOSP = ST_IN_PARENS_NOSP_NO_IDI; + this.IN_PARENS_W_SP = ST_IN_PARENS_W_SP_NO_IDI; + this.IN_MATCHED_PARENS_NOSP = ST_IN_MATCHED_PARENS_NOSP_NO_IDI; + this.IN_MATCHED_PARENS_W_SP = ST_IN_MATCHED_PARENS_W_SP_NO_IDI; + this.IN_BRACES_W_SP = ST_IN_BRACES_W_SP_NO_IDI; + this.DECLARATION = ST_DECLARATION_NO_IDI; + this.ENTITY = ST_ENTITY_NO_IDI; + this.TAGNAME = ST_TAGNAME_NO_IDI; + this.ATTRIBUTENAME = ST_ATTRIBUTENAME_NO_IDI; + this.UNQUOTEDVALUE = ST_UNQUOTEDVALUE_NO_IDI; + this.ATTRIBUTEVALUE = ST_ATTRIBUTEVALUE_NO_IDI; + this.ATTRIBUTEVALUESPEC = ST_ATTRIBUTEVALUESPEC_NO_IDI; + this.ATTRIBUTE = ST_ATTRIBUTE_NO_IDI; + this.OPENTAG = allowNameSpace ? ST_NS_OPENTAG_NO_IDI : ST_OPENTAG_NO_IDI; + this.CLOSETAG = allowNameSpace ? ST_NS_CLOSETAG_NO_IDI : ST_CLOSETAG_NO_IDI; } - // make sure this is consistent with lists settings - this.CODE_BLOCK_INDENT = Parser.CODE_BLOCK_INDENT.get(options); + // init flag based patterns + this.LINK_DESTINATION_ANGLES = spaceInLinkUrl ? ST_LINK_DESTINATION_ANGLES_SPC : ST_LINK_DESTINATION_ANGLES_NO_SPC; + this.ENTITY_HERE = intellijDummyIdentifier ? ST_ENTITY_HERE_IDI : ST_ENTITY_HERE_NO_IDI; + + // init dynamic patterns + synchronized (cachedPatterns) { + this.LINK_DESTINATION_MATCHED_PARENS_NOSP = getCachedPattern("LINK_DESTINATION_MATCHED_PARENS_NOSP", patternTypeFlags.withJekyllMacroInLinkUrl(), entry -> Pattern.compile( + "^(?:" + (parseJekyllMacroInLinkUrl ? IN_BRACES_W_SP + "|" : "") + + (REG_CHAR + "|") + + ESCAPED_CHAR + "|\\\\|\$|\$)*")); + + this.LINK_DESTINATION = getCachedPattern("LINK_DESTINATION", patternTypeFlags.withJekyllMacroSpaceInLinkUrl(), entry -> Pattern.compile( + "^(?:" + (parseJekyllMacroInLinkUrl ? IN_BRACES_W_SP + "|" : "") + + (spaceInLinkUrl ? "(?:" + REG_CHAR_SP + ")|" : REG_CHAR + "|") + + ESCAPED_CHAR + "|\\\\|" + (spaceInLinkUrl ? IN_PARENS_W_SP : IN_PARENS_NOSP) + ")*")); + + this.LINK_DESTINATION_MATCHED_PARENS = getCachedPattern("LINK_DESTINATION_MATCHED_PARENS", patternTypeFlags.withJekyllMacroSpaceInLinkUrl(), entry -> Pattern.compile( + "^(?:" + (parseJekyllMacroInLinkUrl ? IN_BRACES_W_SP + "|" : "") + + (spaceInLinkUrl ? "(?:" + REG_CHAR_SP + ")|" : REG_CHAR + "|") + + ESCAPED_CHAR + "|\\\\|\$|\$)*")); + + this.EMAIL_AUTOLINK = getCachedPattern("EMAIL_AUTOLINK", patternTypeFlags.withHtmlTranslator(), entry -> Pattern.compile( + "^<(" + + "(?:[a-zA-Z0-9" + ADDITIONAL_CHARS + ".!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9" + ADDITIONAL_CHARS + "](?:[a-zA-Z0-9" + ADDITIONAL_CHARS + "-]{0,61}[a-zA-Z0-9" + ADDITIONAL_CHARS + "])?(?:\\.[a-zA-Z0-9" + ADDITIONAL_CHARS + "](?:[a-zA-Z0-9" + ADDITIONAL_CHARS + "-]{0,61}[a-zA-Z0-9" + ADDITIONAL_CHARS + "])?)*)" + + (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + + ")>")); + + this.AUTOLINK = getCachedPattern("AUTOLINK", patternTypeFlags.withHtmlTranslator(), entry -> Pattern.compile( + "^<(" + + "(?:[a-zA-Z][a-zA-Z0-9" + ADDITIONAL_CHARS + ".+-]{1,31}:[^<>" + EXCLUDED_0_TO_SPACE + "]*)" + + (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + + ")>")); + + this.WWW_AUTOLINK = getCachedPattern("WWW_AUTOLINK", patternTypeFlags.withHtmlTranslator(), entry -> Pattern.compile( + "^<(" + + "(?:w" + ADDITIONAL_CHARS + "?){3,3}\\.[^<>" + EXCLUDED_0_TO_SPACE + "]*" + + (htmlForTranslator ? "|(?:" + translationAutolinkTagPattern + ")" : "") + + ")>")); + + this.HTML_TAG = getCachedPattern("HTML_TAG", patternTypeFlags.withHtmlTranslator(), entry -> Pattern.compile('^' + ("(?:" + OPENTAG + "|" + CLOSETAG + "|" + HTMLCOMMENT + + "|" + PROCESSINGINSTRUCTION + "|" + DECLARATION + "|" + CDATA + + (htmlForTranslator ? "|<(?:" + translationHtmlInlineTagPattern + ")>|" : "") + ")"), Pattern.CASE_INSENSITIVE)); + + this.LIST_ITEM_MARKER = getCachedPattern("LIST_ITEM_MARKER", patternTypeFlags.withItemPrefixChars(), entry -> { + if (listsItemMarkerSpace) { + if (listsOrderedItemDotOnly) { + return Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?=[ \t])|^(\\d{1,9})([.])(?=[ \t])"); + } else { + return Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?=[ \t])|^(\\d{1,9})([.)])(?=[ \t])"); + } + } else { + if (listsOrderedItemDotOnly) { + return Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?= |\t|$)|^(\\d{1,9})([.])(?= |\t|$)"); + } else { + return Pattern.compile("^([\\Q" + itemPrefixChars + "\\E])(?= |\t|$)|^(\\d{1,9})([.)])(?= |\t|$)"); + } + } + }); + } - // list of characters not allowed in link URL - this.INVALID_LINK_CHARS = " \t"; + this.HTMLTAG = this.HTML_TAG.pattern(); } + /** + * @deprecated in version (0.62.2), to be removed + */ + @Deprecated public String EXCLUDED_0_TO_SPACE() { - // NOTE: hardcoded because of dependency on it being '\u001f' - return intellijDummyIdentifier ? "\u0000-\u001e\u0020" : "\u0000-\u0020"; + return intellijDummyIdentifier ? ST_EXCLUDED_0_TO_SPACE_IDI : ST_EXCLUDED_0_TO_SPACE_NO_IDI; } + /** + * @deprecated in version (0.62.2), to be removed + */ + @Deprecated public String ADDITIONAL_CHARS() { - // NOTE: hardcoded because of dependency on it being '\u001f' - return intellijDummyIdentifier ? "\u001f" : ""; + return intellijDummyIdentifier ? ST_ADDITIONAL_CHARS_IDI : ST_ADDITIONAL_CHARS_NO_IDI; } + /** + * @deprecated in version (0.62.2), to be removed + */ + @Deprecated public String ADDITIONAL_CHARS_SET(String quantifier) { - // NOTE: hardcoded because of dependency on it being '\u001f' - return intellijDummyIdentifier ? "[\u001f]" + quantifier : ""; + return intellijDummyIdentifier ? ST_ADDITIONAL_CHARS_SET_IDI + quantifier : ST_ADDITIONAL_CHARS_SET_NO_IDI; } public static int columnsToNextTabStop(int column) { @@ -263,18 +492,11 @@ public static int columnsToNextTabStop(int column) { } public static int findLineBreak(CharSequence s, int startIndex) { - for (int i = startIndex; i < s.length(); i++) { - switch (s.charAt(i)) { - case '\n': - case '\r': - return i; - } - } - return -1; + return SequenceUtils.indexOfAny(s, CharPredicate.ANY_EOL, startIndex); } public static boolean isBlank(CharSequence s) { - return findNonSpace(s, 0) == -1; + return SequenceUtils.indexOfAnyNot(s, CharPredicate.BLANKSPACE) == -1; } public static boolean isLetter(CharSequence s, int index) { @@ -283,30 +505,6 @@ public static boolean isLetter(CharSequence s, int index) { } public static boolean isSpaceOrTab(CharSequence s, int index) { - if (index < s.length()) { - switch (s.charAt(index)) { - case ' ': - case '\t': - return true; - } - } - return false; - } - - private static int findNonSpace(CharSequence s, int startIndex) { - for (int i = startIndex; i < s.length(); i++) { - switch (s.charAt(i)) { - case ' ': - case '\t': - case '\n': - case '\u000B': - case '\f': - case '\r': - break; - default: - return i; - } - } - return -1; + return CharPredicate.SPACE_TAB.test(SequenceUtils.safeCharAt(s, index)); } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/formatter/Formatter.java b/flexmark/src/main/java/com/vladsch/flexmark/formatter/Formatter.java index 46206f633..5d59a4598 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/formatter/Formatter.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/formatter/Formatter.java @@ -12,7 +12,6 @@ import com.vladsch.flexmark.util.collection.SubClassingBag; import com.vladsch.flexmark.util.data.*; import com.vladsch.flexmark.util.dependency.DependencyResolver; -import com.vladsch.flexmark.util.dependency.FlatDependencyHandler; import com.vladsch.flexmark.util.format.*; import com.vladsch.flexmark.util.format.options.*; import com.vladsch.flexmark.util.html.Attributes; @@ -32,6 +31,7 @@ import java.util.regex.Pattern; import static com.vladsch.flexmark.formatter.FormattingPhase.DOCUMENT; +import static com.vladsch.flexmark.util.html.Attribute.TITLE_ATTR; /** * Renders a tree of nodes to Markdown. @@ -164,6 +164,10 @@ public class Formatter implements IRender { */ @Deprecated final public static DataKey FORMAT_TABLE_INDENT_PREFIX = TableFormatOptions.FORMAT_TABLE_INDENT_PREFIX; + // used during translation + final public static DataKey> UNIQUIFICATION_MAP = new DataKey<>("REFERENCES_UNIQUIFICATION_MAP", HashMap::new); + final public static DataKey> ATTRIBUTE_UNIQUIFICATION_ID_MAP = new DataKey<>("ATTRIBUTE_UNIQUIFICATION_ID_MAP", HashMap::new); + final private DataHolder options; final List linkResolverFactories; final List nodeFormatterFactories; @@ -173,7 +177,7 @@ public class Formatter implements IRender { this.options = builder.toImmutable(); this.idGeneratorFactory = builder.htmlIdGeneratorFactory == null ? new HeaderIdGenerator.Factory() : builder.htmlIdGeneratorFactory; - this.linkResolverFactories = FlatDependencyHandler.computeDependencies(builder.linkResolverFactories); + this.linkResolverFactories = DependencyResolver.resolveFlatDependencies(builder.linkResolverFactories, null, null); this.nodeFormatterFactories = calculateNodeFormatterFactories(builder.nodeFormatterFactories); } @@ -181,7 +185,6 @@ private static List calculateNodeFormatterFactories(List list = new ArrayList<>(formatterFactories); list.add(new CoreNodeFormatter.Factory()); - return DependencyResolver.resolveFlatDependencies(list, null, null); } @@ -492,6 +495,22 @@ public Builder htmlIdGeneratorFactory(@NotNull HeaderIdGeneratorFactory htmlIdGe addExtensionApiPoint(htmlIdGeneratorFactory); return this; } + + /** + * Add a factory for instantiating a node renderer (done when rendering). This allows to override the rendering + * of node types or define rendering for custom node types. + * + * If multiple node renderers for the same node type are created, the one from the factory that was added first + * "wins". (This is how the rendering for core node types can be overridden; the default rendering comes last.) + * + * @param linkResolverFactory the factory for creating a node renderer + * @return {@code this} + */ + public @NotNull Builder linkResolverFactory(@NotNull LinkResolverFactory linkResolverFactory) { + this.linkResolverFactories.add(linkResolverFactory); + addExtensionApiPoint(linkResolverFactory); + return this; + } } /** @@ -663,12 +682,6 @@ public String encodeUrl(@NotNull CharSequence url) { return String.valueOf(url); } - @NotNull - @Override - public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, Boolean urlEncode) { - return resolveLink(this, linkType, url, null); - } - @NotNull @Override public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, Attributes attributes, Boolean urlEncode) { @@ -692,7 +705,6 @@ ResolvedLink resolveLink(NodeFormatterSubContext context, LinkType linkType, Cha } } - // put it in the map resolvedLinks.put(urlSeq, resolvedLink); } @@ -815,9 +827,8 @@ public Node getCurrentNode() { return renderingNode; } - @NotNull @Override - public DataHolder getOptions() { + public @NotNull DataHolder getOptions() { return options; } @@ -894,17 +905,17 @@ final public Iterable reversedNodesOfType(@NotNull Collection builder) { + public @NotNull NodeFormatterContext getSubContext(@Nullable DataHolder options, @NotNull ISequenceBuilder builder) { return getSubContextRaw(options, builder); } @@ -1095,9 +1106,8 @@ final public Iterable reversedNodesOfType(@NotNull Collection builder) { + public @NotNull NodeFormatterContext getSubContext(@Nullable DataHolder options, @NotNull ISequenceBuilder builder) { MarkdownWriter htmlWriter = new MarkdownWriter(builder, this.markdown.getOptions()); htmlWriter.setContext(this); //noinspection ReturnOfInnerClass diff --git a/flexmark/src/main/java/com/vladsch/flexmark/formatter/FormatterUtils.java b/flexmark/src/main/java/com/vladsch/flexmark/formatter/FormatterUtils.java index b61e23441..6d0a10811 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/formatter/FormatterUtils.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/formatter/FormatterUtils.java @@ -596,7 +596,7 @@ public static void renderTextBlockParagraphLines(Node node, NodeFormatterContext BasedSequence wrappedText = formatter.wrapText().toMapped(SpaceMapper.fromNonBreakSpace); int startLine = markdown.getLineCount(); int firstLineOffset = markdown.column(); - markdown.append(wrappedText).line(); + markdown.pushOptions().preserveSpaces().append(wrappedText).line().popOptions(); if (!paragraphTrackedOffsets.isEmpty()) { LineInfo startLineInfo = markdown.getLineInfo(startLine); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/formatter/internal/CoreNodeFormatter.java b/flexmark/src/main/java/com/vladsch/flexmark/formatter/internal/CoreNodeFormatter.java index 3f008da02..4c56e9aa2 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/formatter/internal/CoreNodeFormatter.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/formatter/internal/CoreNodeFormatter.java @@ -2,6 +2,7 @@ import com.vladsch.flexmark.ast.*; import com.vladsch.flexmark.ast.util.ReferenceRepository; +import com.vladsch.flexmark.formatter.Formatter; import com.vladsch.flexmark.formatter.*; import com.vladsch.flexmark.html.renderer.HtmlIdGenerator; import com.vladsch.flexmark.html.renderer.LinkType; @@ -38,8 +39,18 @@ @SuppressWarnings("WeakerAccess") public class CoreNodeFormatter extends NodeRepositoryFormatter { - final public static DataKey> UNIQUIFICATION_MAP = new DataKey<>("REFERENCES_UNIQUIFICATION_MAP", HashMap::new); - final public static DataKey> ATTRIBUTE_UNIQUIFICATION_ID_MAP = new DataKey<>("ATTRIBUTE_UNIQUIFICATION_ID_MAP", HashMap::new); + /** + * 2020-04-10 + * + * @deprecated use {@link Formatter#UNIQUIFICATION_MAP} + */ + @Deprecated final public static DataKey> UNIQUIFICATION_MAP = Formatter.UNIQUIFICATION_MAP; + /** + * 2020-04-10 + * + * @deprecated use {@link Formatter#ATTRIBUTE_UNIQUIFICATION_ID_MAP} + */ + @Deprecated final public static DataKey> ATTRIBUTE_UNIQUIFICATION_ID_MAP = Formatter.ATTRIBUTE_UNIQUIFICATION_ID_MAP; public static class Factory implements NodeFormatterFactory { @NotNull @@ -59,7 +70,7 @@ public NodeFormatter create(@NotNull DataHolder options) { private Map attributeUniquificationIdMap; public CoreNodeFormatter(DataHolder options) { - super(options, null, UNIQUIFICATION_MAP); + super(options, null, Formatter.UNIQUIFICATION_MAP); formatterOptions = new FormatterOptions(options); this.listOptions = ListOptions.get(options); blankLines = 0; @@ -216,10 +227,10 @@ public void renderReferenceBlock(Reference node, NodeFormatterContext context, M public void renderDocument(@NotNull NodeFormatterContext context, @NotNull MarkdownWriter markdown, @NotNull Document document, @NotNull FormattingPhase phase) { super.renderDocument(context, markdown, document, phase); - attributeUniquificationIdMap = ATTRIBUTE_UNIQUIFICATION_ID_MAP.get(context.getTranslationStore()); + attributeUniquificationIdMap = Formatter.ATTRIBUTE_UNIQUIFICATION_ID_MAP.get(context.getTranslationStore()); if (phase == DOCUMENT_BOTTOM) { - if (context.getFormatterOptions().appendTransferredReferences) { + if (formatterOptions.appendTransferredReferences) { // we will transfer all references which were not part of our document ArrayList> keys = new ArrayList<>(); @@ -276,7 +287,7 @@ private void render(Node node, NodeFormatterContext context, MarkdownWriter mark } } } else { - if (context.getFormatterOptions().keepSoftLineBreaks) { + if (formatterOptions.keepSoftLineBreaks) { markdown.append(chars); } else { markdown.append(FormatterUtils.stripSoftLineBreak(chars, " ")); @@ -290,7 +301,7 @@ private void render(BlankLine node, NodeFormatterContext context, MarkdownWriter blankLines = 0; } blankLines++; - if (blankLines <= context.getFormatterOptions().maxBlankLines) markdown.blankLine(blankLines); + if (blankLines <= formatterOptions.maxBlankLines) markdown.blankLine(blankLines); } } @@ -303,7 +314,6 @@ private void render(Document node, NodeFormatterContext context, MarkdownWriter private void render(Heading node, NodeFormatterContext context, MarkdownWriter markdown) { markdown.blankLine(); HeadingStyle headingPreference = formatterOptions.headingStyle; - FormatterOptions formatterOptions = context.getFormatterOptions(); if (context.isTransformingText() || headingPreference.isNoChange(node.isSetextHeading(), node.getLevel())) { if (node.isAtxHeading()) { markdown.append(node.getOpeningMarker()); @@ -406,8 +416,8 @@ private void render(BlockQuote node, NodeFormatterContext context, MarkdownWrite private void render(ThematicBreak node, NodeFormatterContext context, MarkdownWriter markdown) { markdown.blankLine(); - if (context.getFormatterOptions().thematicBreak != null) { - markdown.append(context.getFormatterOptions().thematicBreak); + if (formatterOptions.thematicBreak != null) { + markdown.append(formatterOptions.thematicBreak); } else { markdown.append(node.getChars()); } @@ -424,7 +434,7 @@ private void render(FencedCodeBlock node, NodeFormatterContext context, Markdown int openingMarkerLen = openingMarker.length(); int closingMarkerLen = closingMarker.length(); - switch (context.getFormatterOptions().fencedCodeMarkerType) { + switch (formatterOptions.fencedCodeMarkerType) { case ANY: break; case BACK_TICK: @@ -437,14 +447,14 @@ private void render(FencedCodeBlock node, NodeFormatterContext context, Markdown break; } - if (openingMarkerLen < context.getFormatterOptions().fencedCodeMarkerLength) openingMarkerLen = context.getFormatterOptions().fencedCodeMarkerLength; - if (closingMarkerLen < context.getFormatterOptions().fencedCodeMarkerLength) closingMarkerLen = context.getFormatterOptions().fencedCodeMarkerLength; + if (openingMarkerLen < formatterOptions.fencedCodeMarkerLength) openingMarkerLen = formatterOptions.fencedCodeMarkerLength; + if (closingMarkerLen < formatterOptions.fencedCodeMarkerLength) closingMarkerLen = formatterOptions.fencedCodeMarkerLength; openingMarker = RepeatedSequence.repeatOf(String.valueOf(openingMarkerChar), openingMarkerLen); - if (context.getFormatterOptions().fencedCodeMatchClosingMarker || closingMarkerChar == SequenceUtils.NUL) { closingMarker = openingMarker; } else closingMarker = RepeatedSequence.repeatOf(String.valueOf(closingMarkerChar), closingMarkerLen); + if (formatterOptions.fencedCodeMatchClosingMarker || closingMarkerChar == SequenceUtils.NUL) { closingMarker = openingMarker; } else closingMarker = RepeatedSequence.repeatOf(String.valueOf(closingMarkerChar), closingMarkerLen); markdown.append(openingMarker); - if (context.getFormatterOptions().fencedCodeSpaceBeforeInfo) markdown.append(' '); + if (formatterOptions.fencedCodeSpaceBeforeInfo) markdown.append(' '); markdown.appendNonTranslating(node.getInfo()); markdown.line(); @@ -452,7 +462,7 @@ private void render(FencedCodeBlock node, NodeFormatterContext context, Markdown if (context.isTransformingText()) { markdown.appendNonTranslating(node.getContentChars()); } else { - if (context.getFormatterOptions().fencedCodeMinimizeIndent) { + if (formatterOptions.fencedCodeMinimizeIndent) { List lines = node.getContentLines(); int[] leadColumns = new int[lines.size()]; int minSpaces = Integer.MAX_VALUE; @@ -503,7 +513,7 @@ private void render(IndentedCodeBlock node, NodeFormatterContext context, Markdo } else { String prefix = RepeatedSequence.repeatOf(" ", listOptions.getCodeIndent()).toString(); - if (context.getFormatterOptions().emulationProfile == ParserEmulationProfile.GITHUB_DOC) { + if (formatterOptions.emulationProfile == ParserEmulationProfile.GITHUB_DOC) { if (node.getParent() instanceof ListItem) { BasedSequence marker = ((ListItem) node.getParent()).getOpeningMarker(); prefix = RepeatedSequence.repeatOf(" ", Utils.minLimit(8 - marker.length() - 1, 4)).toString(); @@ -513,7 +523,7 @@ private void render(IndentedCodeBlock node, NodeFormatterContext context, Markdo markdown.pushPrefix().addPrefix(prefix); markdown.openPreFormatted(true); - if (context.getFormatterOptions().indentedCodeMinimizeIndent) { + if (formatterOptions.indentedCodeMinimizeIndent) { List lines = node.getContentLines(); int[] leadColumns = new int[lines.size()]; int minSpaces = Integer.MAX_VALUE; @@ -635,7 +645,7 @@ private void render(Paragraph node, NodeFormatterContext context, MarkdownWriter } private void render(SoftLineBreak node, NodeFormatterContext context, MarkdownWriter markdown) { - if (context.getFormatterOptions().keepSoftLineBreaks) { + if (formatterOptions.keepSoftLineBreaks || formatterOptions.rightMargin > 0) { markdown.append(node.getChars()); } else if (!markdown.isPendingSpace()) { // need to add a space @@ -644,7 +654,7 @@ private void render(SoftLineBreak node, NodeFormatterContext context, MarkdownWr } private void render(HardLineBreak node, NodeFormatterContext context, MarkdownWriter markdown) { - if (context.getFormatterOptions().keepHardLineBreaks) { + if (formatterOptions.keepHardLineBreaks) { if (context.getRenderPurpose() == FORMAT) { markdown.append(node.getChars()); } else { @@ -667,7 +677,7 @@ private void render(HtmlEntity node, NodeFormatterContext context, MarkdownWrite } private void render(Text node, NodeFormatterContext context, MarkdownWriter markdown) { - if (context.getFormatterOptions().keepSoftLineBreaks) { + if (formatterOptions.keepSoftLineBreaks) { markdown.append(node.getChars()); } else { markdown.append(FormatterUtils.stripSoftLineBreak(node.getChars(), " ")); @@ -681,14 +691,14 @@ private void render(TextBase node, NodeFormatterContext context, MarkdownWriter private void render(Code node, NodeFormatterContext context, MarkdownWriter markdown) { markdown.append(node.getOpeningMarker()); if (context.isTransformingText() || formatterOptions.rightMargin == 0) { - if (context.getFormatterOptions().keepSoftLineBreaks) { + if (formatterOptions.keepSoftLineBreaks) { markdown.appendNonTranslating(node.getText()); } else { markdown.appendNonTranslating(FormatterUtils.stripSoftLineBreak(node.getText(), " ")); } } else { // wrapping text - if (context.getFormatterOptions().keepSoftLineBreaks) { + if (formatterOptions.keepSoftLineBreaks) { markdown.append(node.getText()); } else { markdown.append(FormatterUtils.stripSoftLineBreak(node.getText(), " ")); @@ -863,13 +873,13 @@ private void renderAutoLink(DelimitedLinkNode node, NodeFormatterContext context } private void render(Image node, NodeFormatterContext context, MarkdownWriter markdown) { - if (!context.isTransformingText() && context.getFormatterOptions().rightMargin > 0 && context.getFormatterOptions().keepImageLinksAtStart) { + if (!context.isTransformingText() && formatterOptions.rightMargin > 0 && formatterOptions.keepImageLinksAtStart) { markdown.append(SequenceUtils.LS); } else { - markdown.lineIf(context.getFormatterOptions().keepImageLinksAtStart); + markdown.lineIf(formatterOptions.keepImageLinksAtStart); } - if (!context.getFormatterOptions().optimizedInlineRendering || context.isTransformingText()) { + if (!formatterOptions.optimizedInlineRendering || context.isTransformingText()) { markdown.append(node.getTextOpeningMarker()); if (!context.isTransformingText() || node.getFirstChildAny(HtmlInline.class) != null) { if (formatterOptions.rightMargin > 0) { @@ -897,6 +907,52 @@ private void render(Image node, NodeFormatterContext context, MarkdownWriter mar markdown.append(node.getUrlClosingMarker()); + if (!node.getUrlContent().isEmpty()) { + markdown.openPreFormatted(true); + markdown.pushOptions().preserveSpaces(); + + if (!context.isTransformingText() && formatterOptions.rightMargin > 0) { + CharSequence chars = node.getUrlContent(); + int iMax = chars.length(); + boolean hadEOL = true; + markdown.append('\n'); + + for (int i = 0; i < iMax; i++) { + char c = chars.charAt(i); + + switch (c) { + case '\n': + case '\r': + hadEOL = true; + markdown.append(chars.subSequence(i, i + 1)); + break; + + case ' ': + if (hadEOL) { + markdown.append(BasedSequence.LS); + hadEOL = false; + } + markdown.append(BasedSequence.NBSP); + break; + + default: + if (hadEOL) { + markdown.append(BasedSequence.LS); + hadEOL = false; + } + markdown.append(chars.subSequence(i, i + 1)); + break; + } + } + } else { + markdown.append(node.getUrlContent()); + } + markdown.popOptions(); + markdown.closePreFormatted(); + // title and closing must be first on the line + markdown.append(BasedSequence.LS); + } + if (node.getTitleOpeningMarker().isNotNull()) { markdown.append(' '); markdown.append(node.getTitleOpeningMarker()); @@ -911,13 +967,13 @@ private void render(Image node, NodeFormatterContext context, MarkdownWriter mar } private void render(Link node, NodeFormatterContext context, MarkdownWriter markdown) { - if (!context.isTransformingText() && context.getFormatterOptions().rightMargin > 0 && context.getFormatterOptions().keepExplicitLinksAtStart) { + if (!context.isTransformingText() && formatterOptions.rightMargin > 0 && formatterOptions.keepExplicitLinksAtStart) { markdown.append(SequenceUtils.LS); } else { - markdown.lineIf(context.getFormatterOptions().keepExplicitLinksAtStart); + markdown.lineIf(formatterOptions.keepExplicitLinksAtStart); } - if (!context.getFormatterOptions().optimizedInlineRendering || context.isTransformingText()) { + if (!formatterOptions.optimizedInlineRendering || context.isTransformingText()) { markdown.append(node.getTextOpeningMarker()); if (!context.isTransformingText() || node.getFirstChildAny(HtmlInline.class) != null) { if (formatterOptions.rightMargin > 0) { @@ -979,7 +1035,7 @@ private void render(Link node, NodeFormatterContext context, MarkdownWriter mark } private void render(ImageRef node, NodeFormatterContext context, MarkdownWriter markdown) { - if (!context.getFormatterOptions().optimizedInlineRendering || context.isTransformingText()) { + if (!formatterOptions.optimizedInlineRendering || context.isTransformingText()) { if (context.isTransformingText() || formatterOptions.rightMargin == 0) { if (node.isReferenceTextCombined()) { markdown.append(node.getReferenceOpeningMarker()); @@ -1000,7 +1056,11 @@ private void render(ImageRef node, NodeFormatterContext context, MarkdownWriter } else { if (node.isReferenceTextCombined()) { markdown.append(node.getReferenceOpeningMarker()); - markdown.append(node.getReference().toMapped(SpaceMapper.toNonBreakSpace)); + if (node.isOrDescendantOfType(Paragraph.class)) { + markdown.append(node.getReference().toMapped(SpaceMapper.toNonBreakSpace)); + } else { + markdown.append(node.getReference()); + } markdown.append(node.getReferenceClosingMarker()); markdown.append(node.getTextOpeningMarker()); @@ -1021,7 +1081,7 @@ private void render(ImageRef node, NodeFormatterContext context, MarkdownWriter } private void render(LinkRef node, NodeFormatterContext context, MarkdownWriter markdown) { - if (!context.getFormatterOptions().optimizedInlineRendering || context.isTransformingText()) { + if (!formatterOptions.optimizedInlineRendering || context.isTransformingText()) { if (context.isTransformingText() || formatterOptions.rightMargin == 0) { if (node.isReferenceTextCombined()) { markdown.append(node.getReferenceOpeningMarker()); @@ -1048,7 +1108,11 @@ private void render(LinkRef node, NodeFormatterContext context, MarkdownWriter m } else { if (node.isReferenceTextCombined()) { markdown.append(node.getReferenceOpeningMarker()); - markdown.append(node.getReference().toMapped(SpaceMapper.toNonBreakSpace)); + if (node.isOrDescendantOfType(Paragraph.class)) { + markdown.append(node.getReference().toMapped(SpaceMapper.toNonBreakSpace)); + } else { + markdown.append(node.getReference()); + } markdown.append(node.getReferenceClosingMarker()); markdown.append(node.getTextOpeningMarker()); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/AttributeProvider.java b/flexmark/src/main/java/com/vladsch/flexmark/html/AttributeProvider.java index 9ae103477..f9ab8f51b 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/AttributeProvider.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/AttributeProvider.java @@ -3,6 +3,7 @@ import com.vladsch.flexmark.html.renderer.AttributablePart; import com.vladsch.flexmark.util.ast.Node; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import org.jetbrains.annotations.NotNull; /** @@ -26,11 +27,10 @@ public interface AttributeProvider { * Core defines LinkStatus.UNKNOWN,LinkStatus.VALID,LinkStatus.NOT_FOUND. Extensions can define more. * * AttributablePart.NODE is a generic placeholder when the node did not provide a specific part for attribution. - * - * @param node the node to set attributes for + * @param node the node to set attributes for * @param part attributes for the specific part of the node being generated, Core defines AttributablePart.LINK, * AttributablePart.ID and generic AttributablePart.NODE, extensions are free to define more * @param attributes the attributes, with any default attributes already set in the map */ - void setAttributes(@NotNull Node node, @NotNull AttributablePart part, @NotNull Attributes attributes); + void setAttributes(@NotNull Node node, @NotNull AttributablePart part, @NotNull MutableAttributes attributes); } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/EmbeddedAttributeProvider.java b/flexmark/src/main/java/com/vladsch/flexmark/html/EmbeddedAttributeProvider.java index 627a50747..d727a07a1 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/EmbeddedAttributeProvider.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/EmbeddedAttributeProvider.java @@ -4,6 +4,7 @@ import com.vladsch.flexmark.html.renderer.LinkResolverContext; import com.vladsch.flexmark.util.ast.Node; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import com.vladsch.flexmark.util.sequence.BasedSequence; import org.jetbrains.annotations.NotNull; @@ -24,7 +25,7 @@ public AttributeProvider apply(@NotNull LinkResolverContext context) { } @Override - public void setAttributes(@NotNull Node node, @NotNull AttributablePart part, @NotNull Attributes attributes) { + public void setAttributes(@NotNull Node node, @NotNull AttributablePart part, @NotNull MutableAttributes attributes) { if (part == AttributablePart.NODE) { Node firstChild = node.getChildOfType(EmbeddedNodeAttributes.class); if (firstChild instanceof EmbeddedNodeAttributes) { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRenderer.java b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRenderer.java index 084b34001..91a9db8c5 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRenderer.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRenderer.java @@ -12,6 +12,7 @@ import com.vladsch.flexmark.util.format.TrackedOffset; import com.vladsch.flexmark.util.format.TrackedOffsetUtils; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import com.vladsch.flexmark.util.misc.Extension; import com.vladsch.flexmark.util.misc.Pair; import com.vladsch.flexmark.util.sequence.Escaping; @@ -61,11 +62,16 @@ public class HtmlRenderer implements IRender { final public static DataKey HEADER_ID_GENERATOR_NON_DASH_CHARS = SharedDataKeys.HEADER_ID_GENERATOR_NON_DASH_CHARS; final public static DataKey HEADER_ID_GENERATOR_NO_DUPED_DASHES = SharedDataKeys.HEADER_ID_GENERATOR_NO_DUPED_DASHES; final public static DataKey HEADER_ID_GENERATOR_NON_ASCII_TO_LOWERCASE = SharedDataKeys.HEADER_ID_GENERATOR_NON_ASCII_TO_LOWERCASE; + final public static DataKey HEADER_ID_REF_TEXT_TRIM_LEADING_SPACES = SharedDataKeys.HEADER_ID_REF_TEXT_TRIM_LEADING_SPACES; + final public static DataKey HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES = SharedDataKeys.HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES; + final public static DataKey HEADER_ID_ADD_EMOJI_SHORTCUT = SharedDataKeys.HEADER_ID_ADD_EMOJI_SHORTCUT; final public static DataKey RENDER_HEADER_ID = SharedDataKeys.RENDER_HEADER_ID; final public static DataKey GENERATE_HEADER_ID = SharedDataKeys.GENERATE_HEADER_ID; final public static DataKey DO_NOT_RENDER_LINKS = SharedDataKeys.DO_NOT_RENDER_LINKS; - final public static DataKey FENCED_CODE_LANGUAGE_CLASS_PREFIX = new DataKey<>("FENCED_CODE_LANGUAGE_CLASS_PREFIX", "language-"); + final public static DataKey FENCED_CODE_LANGUAGE_CLASS_PREFIX = new DataKey<>("FENCED_CODE_LANGUAGE_CLASS_PREFIX", "language-"); // prefix to add to unmapped info strings + final public static DataKey> FENCED_CODE_LANGUAGE_CLASS_MAP = new DataKey<>("FENCED_CODE_LANGUAGE_CLASS_MAP", HashMap::new); // info to language class mapping final public static DataKey FENCED_CODE_NO_LANGUAGE_CLASS = new DataKey<>("FENCED_CODE_NO_LANGUAGE_CLASS", ""); + final public static DataKey FENCED_CODE_LANGUAGE_DELIMITERS = new DataKey<>("FENCED_CODE_LANGUAGE_DELIMITERS", " \t"); final public static DataKey SOURCE_POSITION_ATTRIBUTE = new DataKey<>("SOURCE_POSITION_ATTRIBUTE", ""); final public static DataKey SOURCE_POSITION_PARAGRAPH_LINES = new DataKey<>("SOURCE_POSITION_PARAGRAPH_LINES", false); final public static DataKey TYPE = new DataKey<>("TYPE", "HTML"); @@ -430,6 +436,17 @@ public boolean isRendererType(@NotNull String supportedRendererType) { return this; } + /** + * Add a factory for resolving URI to content + * + * @param contentResolverFactory the factory for creating a node renderer + * @return {@code this} + */ + @Override + public @NotNull Builder contentResolverFactory(@NotNull UriContentResolverFactory contentResolverFactory) { + throw new IllegalStateException("Not implemented"); + } + /** * Add a factory for generating the header id attribute from the header's text * @@ -524,7 +541,7 @@ public void dispose() { this.myLinkResolvers = new LinkResolver[linkResolverFactories.size()]; this.doNotRenderLinksNesting = htmlOptions.doNotRenderLinksInDocument ? 0 : 1; this.htmlIdGenerator = htmlIdGeneratorFactory != null ? htmlIdGeneratorFactory.create(this) - : (!(htmlOptions.renderHeaderId || htmlOptions.generateHeaderIds) ? HtmlIdGenerator.NULL : new HeaderIdGenerator.Factory().create(this)); + : (!(htmlOptions.generateHeaderIds) ? HtmlIdGenerator.NULL : new HeaderIdGenerator.Factory().create(this)); htmlWriter.setContext(this); @@ -565,12 +582,6 @@ public Node getCurrentNode() { return renderingNode; } - @NotNull - @Override - public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, Boolean urlEncode) { - return resolveLink(linkType, url, (Attributes) null, urlEncode); - } - @NotNull @Override public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, Attributes attributes, Boolean urlEncode) { @@ -594,7 +605,6 @@ public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequenc } } - // put it in the map resolvedLinks.put(urlSeq, resolvedLink); } @@ -605,7 +615,7 @@ public ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequenc public String getNodeId(@NotNull Node node) { String id = htmlIdGenerator.getId(node); if (attributeProviderFactories.size() != 0) { - Attributes attributes = new Attributes(); + MutableAttributes attributes = new MutableAttributes(); if (id != null) attributes.replaceValue("id", id); for (AttributeProvider attributeProvider : attributeProviders) { @@ -650,20 +660,18 @@ public String encodeUrl(@NotNull CharSequence url) { } } - @NotNull @Override - public Attributes extendRenderingNodeAttributes(@NotNull AttributablePart part, Attributes attributes) { - Attributes attr = attributes != null ? attributes : new Attributes(); + public @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull AttributablePart part, Attributes attributes) { + MutableAttributes attr = attributes != null ? attributes.toMutable() : new MutableAttributes(); for (AttributeProvider attributeProvider : attributeProviders) { attributeProvider.setAttributes(this.renderingNode, part, attr); } return attr; } - @NotNull @Override - public Attributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, Attributes attributes) { - Attributes attr = attributes != null ? attributes : new Attributes(); + public @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, Attributes attributes) { + MutableAttributes attr = attributes != null ? attributes.toMutable() : new MutableAttributes(); for (AttributeProvider attributeProvider : attributeProviders) { attributeProvider.setAttributes(node, part, attr); } @@ -829,18 +837,16 @@ public SubNodeRenderer(MainNodeRenderer mainNodeRenderer, HtmlWriter htmlWriter, @Override public String encodeUrl(@NotNull CharSequence url) {return myMainNodeRenderer.encodeUrl(url);} - @NotNull @Override - public Attributes extendRenderingNodeAttributes(@NotNull AttributablePart part, Attributes attributes) { + public @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull AttributablePart part, Attributes attributes) { return myMainNodeRenderer.extendRenderingNodeAttributes( part, attributes ); } - @NotNull @Override - public Attributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, Attributes attributes) { + public @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, Attributes attributes) { return myMainNodeRenderer.extendRenderingNodeAttributes( node, part, diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRendererOptions.java b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRendererOptions.java index 6be020e65..897e7a6a4 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRendererOptions.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlRendererOptions.java @@ -1,10 +1,12 @@ package com.vladsch.flexmark.html; import com.vladsch.flexmark.util.data.DataHolder; +import com.vladsch.flexmark.util.misc.CharPredicate; import com.vladsch.flexmark.util.misc.Utils; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; +import java.util.HashMap; import java.util.regex.Pattern; public class HtmlRendererOptions { @@ -31,6 +33,9 @@ public class HtmlRendererOptions { final public boolean renderHeaderId; final public boolean generateHeaderIds; final public @NotNull String languageClassPrefix; + final public @NotNull HashMap languageClassMap; + final public @NotNull String languageDelimiters; + final public CharPredicate languageDelimiterSet; final public @NotNull String noLanguageClass; final public @NotNull String sourcePositionAttribute; final public @Nullable String inlineCodeSpliceClass; @@ -70,6 +75,9 @@ public HtmlRendererOptions(DataHolder options) { renderHeaderId = HtmlRenderer.RENDER_HEADER_ID.get(options); generateHeaderIds = HtmlRenderer.GENERATE_HEADER_ID.get(options); languageClassPrefix = HtmlRenderer.FENCED_CODE_LANGUAGE_CLASS_PREFIX.get(options); + languageClassMap = HtmlRenderer.FENCED_CODE_LANGUAGE_CLASS_MAP.get(options); + languageDelimiters = HtmlRenderer.FENCED_CODE_LANGUAGE_DELIMITERS.get(options); + languageDelimiterSet = CharPredicate.anyOf(languageDelimiters); noLanguageClass = HtmlRenderer.FENCED_CODE_NO_LANGUAGE_CLASS.get(options); sourcePositionAttribute = HtmlRenderer.SOURCE_POSITION_ATTRIBUTE.get(options); sourcePositionParagraphLines = !sourcePositionAttribute.isEmpty() && HtmlRenderer.SOURCE_POSITION_PARAGRAPH_LINES.get(options); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlWriter.java b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlWriter.java index 2f047b605..5f6d2081d 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlWriter.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/HtmlWriter.java @@ -15,8 +15,8 @@ import java.util.ArrayList; public class HtmlWriter extends HtmlAppendableBase { - private NodeRendererContext context; - private AttributablePart useAttributes; + private @Nullable NodeRendererContext context; + private @Nullable AttributablePart useAttributes; public HtmlWriter(int indentSize, int formatOptions) { super(indentSize, formatOptions); @@ -42,19 +42,20 @@ void setContext(@NotNull NodeRendererContext context) { } public @NotNull NodeRendererContext getContext() { + assert context != null; return context; } public @NotNull HtmlWriter srcPos() { - return srcPos(context.getCurrentNode().getChars()); + return context == null ? this : srcPos(context.getCurrentNode().getChars()); } public @NotNull HtmlWriter srcPosWithEOL() { - return srcPosWithEOL(context.getCurrentNode().getChars()); + return context == null ? this : srcPosWithEOL(context.getCurrentNode().getChars()); } public @NotNull HtmlWriter srcPosWithTrailingEOL() { - return srcPosWithTrailingEOL(context.getCurrentNode().getChars()); + return context == null ? this : srcPosWithTrailingEOL(context.getCurrentNode().getChars()); } public @NotNull HtmlWriter srcPos(@NotNull BasedSequence sourceText) { @@ -98,7 +99,7 @@ void setContext(@NotNull NodeRendererContext context) { } public @NotNull HtmlWriter srcPos(int startOffset, int endOffset) { - if (startOffset <= endOffset && !context.getHtmlOptions().sourcePositionAttribute.isEmpty()) { + if (startOffset <= endOffset && context != null && !context.getHtmlOptions().sourcePositionAttribute.isEmpty()) { super.attr(context.getHtmlOptions().sourcePositionAttribute, startOffset + "-" + endOffset); } return this; @@ -128,10 +129,18 @@ public HtmlWriter withAttr() { @Override public HtmlWriter tag(@NotNull CharSequence tagName, boolean voidElement) { if (useAttributes != null) { - final Attributes attributes = context.extendRenderingNodeAttributes(useAttributes, getAttributes()); - String sourcePositionAttribute = context.getHtmlOptions().sourcePositionAttribute; - String attributeValue = attributes.getValue(sourcePositionAttribute); - + String attributeValue; + final Attributes attributes; + + if (context != null) { + attributes = context.extendRenderingNodeAttributes(useAttributes, getAttributes()); + String sourcePositionAttribute = context.getHtmlOptions().sourcePositionAttribute; + attributeValue = attributes.getValue(sourcePositionAttribute); + } else { + attributeValue = ""; + attributes = new Attributes(); + } + if (!attributeValue.isEmpty()) { // add to tag ranges int pos = attributeValue.indexOf('-'); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/RendererBuilder.java b/flexmark/src/main/java/com/vladsch/flexmark/html/RendererBuilder.java index 07533f2d5..1519ad530 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/RendererBuilder.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/RendererBuilder.java @@ -16,18 +16,23 @@ public interface RendererBuilder extends DataHolder { * @return {@code this} */ @NotNull RendererBuilder attributeProviderFactory(@NotNull AttributeProviderFactory attributeProviderFactory); + /** - * Add a factory for instantiating a node renderer (done when rendering). This allows to override the rendering - * of node types or define rendering for custom node types. - * - * If multiple node renderers for the same node type are created, the one from the factory that was added first - * "wins". (This is how the rendering for core node types can be overridden; the default rendering comes last.) + * Add a factory for resolving links in markdown to URI used in rendering * * @param linkResolverFactory the factory for creating a node renderer * @return {@code this} */ @NotNull RendererBuilder linkResolverFactory(@NotNull LinkResolverFactory linkResolverFactory); + /** + * Add a factory for resolving URI to content + * + * @param contentResolverFactory the factory for creating a node renderer + * @return {@code this} + */ + @NotNull RendererBuilder contentResolverFactory(@NotNull UriContentResolverFactory contentResolverFactory); + /** * Add a factory for generating the header id attribute from the header's text * diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolver.java b/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolver.java new file mode 100644 index 000000000..3c8a17f3a --- /dev/null +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolver.java @@ -0,0 +1,12 @@ +package com.vladsch.flexmark.html; + +import com.vladsch.flexmark.html.renderer.LinkResolverBasicContext; +import com.vladsch.flexmark.html.renderer.ResolvedContent; +import com.vladsch.flexmark.util.ast.Node; +import org.jetbrains.annotations.NotNull; + +public interface UriContentResolver { + @NotNull ResolvedContent resolveContent(@NotNull Node node, @NotNull LinkResolverBasicContext context, @NotNull ResolvedContent content); + + UriContentResolver NULL = (node, context, content) -> content; +} diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolverFactory.java b/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolverFactory.java new file mode 100644 index 000000000..ac0de0c17 --- /dev/null +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/UriContentResolverFactory.java @@ -0,0 +1,23 @@ +package com.vladsch.flexmark.html; + +import com.vladsch.flexmark.html.renderer.LinkResolverBasicContext; +import com.vladsch.flexmark.util.dependency.Dependent; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + +import java.util.Set; +import java.util.function.Function; + +public interface UriContentResolverFactory extends Function, Dependent { + @Override + @Nullable Set> getAfterDependents(); + + @Override + @Nullable Set> getBeforeDependents(); + + @Override + boolean affectsGlobalScope(); + + @Override + @NotNull UriContentResolver apply(@NotNull LinkResolverBasicContext context); +} diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/CoreNodeRenderer.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/CoreNodeRenderer.java index 9e6a822b0..813a8080b 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/CoreNodeRenderer.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/CoreNodeRenderer.java @@ -117,7 +117,7 @@ void render(Document node, NodeRendererContext context, HtmlWriter html) { void render(Heading node, NodeRendererContext context, HtmlWriter html) { if (context.getHtmlOptions().renderHeaderId) { String id = context.getNodeId(node); - if (id != null) { + if (id != null && id.length() > 0) { html.attr("id", id); } } @@ -143,11 +143,13 @@ void render(FencedCodeBlock node, NodeRendererContext context, HtmlWriter html) html.srcPosWithTrailingEOL(node.getChars()).withAttr().tag("pre").openPre(); BasedSequence info = node.getInfo(); + HtmlRendererOptions htmlOptions = context.getHtmlOptions(); if (info.isNotNull() && !info.isBlank()) { - BasedSequence language = node.getInfoDelimitedByAny(CharPredicate.SPACE_TAB); - html.attr("class", context.getHtmlOptions().languageClassPrefix + language.unescape()); + String language = node.getInfoDelimitedByAny(htmlOptions.languageDelimiterSet).unescape(); + String languageClass = htmlOptions.languageClassMap.getOrDefault(language, htmlOptions.languageClassPrefix + language); + html.attr("class", languageClass); } else { - String noLanguageClass = context.getHtmlOptions().noLanguageClass.trim(); + String noLanguageClass = htmlOptions.noLanguageClass.trim(); if (!noLanguageClass.isEmpty()) { html.attr("class", noLanguageClass); } @@ -161,7 +163,7 @@ void render(FencedCodeBlock node, NodeRendererContext context, HtmlWriter html) } html.tag("/code"); html.tag("/pre").closePre(); - html.lineIf(context.getHtmlOptions().htmlBlockCloseTagEol); + html.lineIf(htmlOptions.htmlBlockCloseTagEol); } @SuppressWarnings("MethodMayBeStatic") @@ -629,12 +631,10 @@ void render(Image node, NodeRendererContext context, HtmlWriter html) { // we have a title part, use that if (node.getTitle().isNotNull()) { - resolvedLink.getNonNullAttributes().replaceValue(Attribute.TITLE_ATTR, node.getTitle().unescape()); - } else { - resolvedLink.getNonNullAttributes().remove(Attribute.TITLE_ATTR); + resolvedLink = resolvedLink.withTitle(node.getTitle().unescape()); } - html.attr(resolvedLink.getAttributes()); + html.attr(resolvedLink.getNonNullAttributes()); html.srcPos(node.getChars()).withAttr(resolvedLink).tagVoid("img"); } } @@ -649,12 +649,10 @@ void render(Link node, NodeRendererContext context, HtmlWriter html) { // we have a title part, use that if (node.getTitle().isNotNull()) { - resolvedLink.getNonNullAttributes().replaceValue(Attribute.TITLE_ATTR, node.getTitle().unescape()); - } else { - resolvedLink.getNonNullAttributes().remove(Attribute.TITLE_ATTR); + resolvedLink = resolvedLink.withTitle(node.getTitle().unescape()); } - html.attr(resolvedLink.getAttributes()); + html.attr(resolvedLink.getNonNullAttributes()); html.srcPos(node.getChars()).withAttr(resolvedLink).tag("a"); renderChildrenSourceLineWrapped(node, node.getText(), context, html); html.tag("/a"); @@ -700,9 +698,7 @@ void render(ImageRef node, NodeRendererContext context, HtmlWriter html) { resolvedLink = context.resolveLink(LinkType.IMAGE, url, null, null); if (reference.getTitle().isNotNull()) { - resolvedLink.getNonNullAttributes().replaceValue(Attribute.TITLE_ATTR, reference.getTitle().unescape()); - } else { - resolvedLink.getNonNullAttributes().remove(Attribute.TITLE_ATTR); + resolvedLink = resolvedLink.withTitle(reference.getTitle().unescape()); } } else { // see if have reference resolver and this is resolved @@ -753,9 +749,7 @@ void render(LinkRef node, NodeRendererContext context, HtmlWriter html) { resolvedLink = context.resolveLink(LinkType.LINK, url, null, null); if (reference.getTitle().isNotNull()) { - resolvedLink.getNonNullAttributes().replaceValue(Attribute.TITLE_ATTR, reference.getTitle().unescape()); - } else { - resolvedLink.getNonNullAttributes().remove(Attribute.TITLE_ATTR); + resolvedLink = resolvedLink.withTitle(reference.getTitle().unescape()); } } else { // see if have reference resolver and this is resolved diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/FileUriContentResolver.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/FileUriContentResolver.java new file mode 100644 index 000000000..021fafdb5 --- /dev/null +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/FileUriContentResolver.java @@ -0,0 +1,70 @@ +package com.vladsch.flexmark.html.renderer; + +import com.vladsch.flexmark.html.UriContentResolver; +import com.vladsch.flexmark.html.UriContentResolverFactory; +import com.vladsch.flexmark.util.ast.Node; +import com.vladsch.flexmark.util.dependency.LastDependent; +import com.vladsch.flexmark.util.misc.FileUtil; +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + +import java.io.File; +import java.io.IOException; +import java.util.Collections; +import java.util.Set; + +public class FileUriContentResolver implements UriContentResolver { + public FileUriContentResolver(LinkResolverBasicContext context) { + + } + + @Override + public @NotNull ResolvedContent resolveContent(@NotNull Node node, @NotNull LinkResolverBasicContext context, @NotNull ResolvedContent content) { + ResolvedLink resolvedLink = content.getResolvedLink(); + + if (resolvedLink.getStatus() == LinkStatus.VALID) { + // have the file + String url = resolvedLink.getUrl(); + if (url.startsWith("file:/")) { + // handle Windows and OSX/Unix URI + String substring = url.startsWith("file://") ? url.substring("file://".length()) : File.separatorChar == '\\' ? url.substring("file:/".length()) : url.substring("file:".length()); + File includedFile = new File(substring); + if (includedFile.isFile() && includedFile.exists()) { + // need to read and parse the file + try { + return content.withContent(FileUtil.getFileContentBytesWithExceptions(includedFile)).withStatus(LinkStatus.VALID); + } catch (IOException e) { + e.printStackTrace(); + } + } + } + } + return content; + } + + public static class Factory implements UriContentResolverFactory { + @Nullable + @Override + public Set> getAfterDependents() { + // ensure that default file uri resolver is the last one in the list + return Collections.singleton(LastDependent.class); + } + + @Nullable + @Override + public Set> getBeforeDependents() { + return null; + } + + @Override + public boolean affectsGlobalScope() { + return false; + } + + @NotNull + @Override + public UriContentResolver apply(@NotNull LinkResolverBasicContext context) { + return new FileUriContentResolver(context); + } + } +} diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/LinkResolverContext.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/LinkResolverContext.java index 430281f7b..be9ed7936 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/LinkResolverContext.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/LinkResolverContext.java @@ -17,15 +17,18 @@ public interface LinkResolverContext extends LinkResolverBasicContext { * @return the current renderer options {@link DataHolder} */ @NotNull DataHolder getOptions(); + /** * @return the {@link Document} node of the current context */ @NotNull Document getDocument(); + /** * @param url to be encoded * @return an encoded URL (depending on the configuration) */ @NotNull String encodeUrl(@NotNull CharSequence url); + /** * Render the specified node and its children using the configured renderers. This should be used to render child * nodes; be careful not to pass the node that is being rendered, that would result in an endless loop. @@ -33,16 +36,19 @@ public interface LinkResolverContext extends LinkResolverBasicContext { * @param node the node to render */ void render(@NotNull Node node); + /** * Render the children of the node, used by custom renderers * * @param parent node the children of which are to be rendered */ void renderChildren(@NotNull Node parent); + /** * @return the current node being rendered */ @NotNull Node getCurrentNode(); + /** * Resolve link for rendering. Link Resolvers are going to be called until one returns ResolvedLink with getStatus() != LinkStatus.Unknown * @@ -56,7 +62,10 @@ public interface LinkResolverContext extends LinkResolverBasicContext { * @param urlEncode whether the link should be url encoded, if null then the value of {@link HtmlRenderer#PERCENT_ENCODE_URLS} will be used to determine whether the resolved URL is to be encoded. * @return resolved link url for this link and its resolved status */ - @NotNull ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, @Nullable Boolean urlEncode); + default @NotNull ResolvedLink resolveLink(@NotNull LinkType linkType, @NotNull CharSequence url, @Nullable Boolean urlEncode) { + return resolveLink(linkType, url, null, urlEncode); + } + /** * Resolve link for rendering. Link Resolvers are going to be called until one returns ResolvedLink with getStatus() != LinkStatus.Unknown * diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/NodeRendererContext.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/NodeRendererContext.java index 8a4229485..c40db09cc 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/NodeRendererContext.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/NodeRendererContext.java @@ -4,6 +4,7 @@ import com.vladsch.flexmark.html.HtmlWriter; import com.vladsch.flexmark.util.ast.Node; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttributes; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; @@ -19,7 +20,7 @@ public interface NodeRendererContext extends LinkResolverContext { * @param attributes the attributes that were calculated by the renderer or null, these may be modified. To preserve originals pass a copy. * @return the extended attributes with added/updated/removed entries */ - @NotNull Attributes extendRenderingNodeAttributes(@NotNull AttributablePart part, @Nullable Attributes attributes); + @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull AttributablePart part, @Nullable Attributes attributes); /** * Extend the attributes by extensions for the node being currently rendered. @@ -29,7 +30,7 @@ public interface NodeRendererContext extends LinkResolverContext { * @param attributes the attributes that were calculated by the renderer or null, these may be modified. To preserve originals pass a copy. * @return the extended attributes with added/updated/removed entries */ - @NotNull Attributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, @Nullable Attributes attributes); + @NotNull MutableAttributes extendRenderingNodeAttributes(@NotNull Node node, @NotNull AttributablePart part, @Nullable Attributes attributes); /** * @return the HTML writer to use diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedContent.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedContent.java new file mode 100644 index 000000000..f19338609 --- /dev/null +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedContent.java @@ -0,0 +1,56 @@ +package com.vladsch.flexmark.html.renderer; + +import org.jetbrains.annotations.NotNull; +import org.jetbrains.annotations.Nullable; + +import java.util.Arrays; + +public class ResolvedContent { + final private @NotNull ResolvedLink resolvedLink; + final private @NotNull LinkStatus status; + final private @Nullable byte[] content; + + public ResolvedContent(@NotNull ResolvedLink resolvedLink, @NotNull LinkStatus status, @Nullable byte[] content) { + this.resolvedLink = resolvedLink; + this.status = status; + this.content = content; + } + + + // @formatter:off + public ResolvedContent withStatus(@NotNull LinkStatus status) { return status == this.status ? this : new ResolvedContent(resolvedLink, status, content); } + public ResolvedContent withContent(@Nullable byte[] content) { return Arrays.equals(this.content, content) ? this : new ResolvedContent(resolvedLink, status, content); } + // @formatter:on + + public @NotNull ResolvedLink getResolvedLink() { + return resolvedLink; + } + + public @NotNull LinkStatus getStatus() { + return status; + } + + public @Nullable byte[] getContent() { + return content; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + + ResolvedContent content1 = (ResolvedContent) o; + + if (!resolvedLink.equals(content1.resolvedLink)) return false; + if (!status.equals(content1.status)) return false; + return Arrays.equals(content, content1.content); + } + + @Override + public int hashCode() { + int result = resolvedLink.hashCode(); + result = 31 * result + status.hashCode(); + result = 31 * result + Arrays.hashCode(content); + return result; + } +} diff --git a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedLink.java b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedLink.java index 640d26b68..f584923c7 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedLink.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/html/renderer/ResolvedLink.java @@ -2,6 +2,8 @@ import com.vladsch.flexmark.util.html.Attribute; import com.vladsch.flexmark.util.html.Attributes; +import com.vladsch.flexmark.util.html.MutableAttribute; +import com.vladsch.flexmark.util.html.MutableAttributes; import org.jetbrains.annotations.NotNull; import org.jetbrains.annotations.Nullable; @@ -9,7 +11,7 @@ public class ResolvedLink { final private @NotNull LinkType myLinkType; final private @NotNull String myUrl; final private @NotNull LinkStatus myStatus; - private @Nullable Attributes myAttributes; + private @Nullable MutableAttributes myAttributes; public ResolvedLink(@NotNull LinkType linkType, @NotNull CharSequence url) { this(linkType, url, null, LinkStatus.UNKNOWN); @@ -21,12 +23,19 @@ public ResolvedLink(@NotNull LinkType linkType, @NotNull CharSequence url, @Null @Nullable public Attributes getAttributes() { - return myAttributes; + return myAttributes == null ? null : myAttributes.toImmutable(); + } + + public @NotNull Attributes getNonNullAttributes() { + if (myAttributes == null) { + myAttributes = new MutableAttributes(); + } + return myAttributes.toImmutable(); } - public @NotNull Attributes getNonNullAttributes() { + public @NotNull MutableAttributes getMutableAttributes() { if (myAttributes == null) { - myAttributes = new Attributes(); + myAttributes = new MutableAttributes(); } return myAttributes; } @@ -36,7 +45,7 @@ public ResolvedLink(@NotNull LinkType linkType, CharSequence url, @Nullable Attr myUrl = String.valueOf(url); myStatus = status; if (attributes != null) { - getNonNullAttributes().addValues(attributes); + getMutableAttributes().addValues(attributes); } } @@ -89,7 +98,7 @@ public String getUrl() { String haveTitle = myAttributes == null ? null : myAttributes.getValue(Attribute.TITLE_ATTR); if (title == haveTitle || haveTitle != null && title != null && haveTitle.contentEquals(title)) return this; - Attributes attributes = new Attributes(myAttributes); + MutableAttributes attributes = new MutableAttributes(myAttributes); if (title == null) { attributes.remove(Attribute.TITLE_ATTR); if (attributes.isEmpty()) attributes = null; @@ -107,7 +116,7 @@ public String getUrl() { String haveTarget = myAttributes == null ? null : myAttributes.getValue(Attribute.TARGET_ATTR); if (target == haveTarget || haveTarget != null && target != null && haveTarget.contentEquals(target)) return this; - Attributes attributes = new Attributes(myAttributes); + MutableAttributes attributes = new MutableAttributes(myAttributes); if (target == null) { attributes.remove(Attribute.TARGET_ATTR); if (attributes.isEmpty()) attributes = null; diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/InlineParserOptions.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/InlineParserOptions.java index bbfc95192..1c8265eee 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/InlineParserOptions.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/InlineParserOptions.java @@ -15,6 +15,7 @@ public class InlineParserOptions { final public boolean intellijDummyIdentifier; final public boolean parseJekyllMacrosInUrls; final public boolean useHardcodedLinkAddressParser; + final public boolean linkTextPriorityOverLinkRef; public InlineParserOptions(DataHolder options) { matchLookaheadFirst = Parser.MATCH_NESTED_LINK_REFS_FIRST.get(options); @@ -29,5 +30,6 @@ public InlineParserOptions(DataHolder options) { codeSoftLineBreaks = Parser.CODE_SOFT_LINE_BREAKS.get(options); inlineDelimiterDirectionalPunctuations = Parser.INLINE_DELIMITER_DIRECTIONAL_PUNCTUATIONS.get(options); linksAllowMatchedParentheses = Parser.LINKS_ALLOW_MATCHED_PARENTHESES.get(options); + linkTextPriorityOverLinkRef = Parser.LINK_TEXT_PRIORITY_OVER_LINK_REF.get(options); } } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/Parser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/Parser.java index 8018f3afb..22a93ce4d 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/Parser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/Parser.java @@ -85,6 +85,7 @@ public class Parser implements IParse { final public static DataKey SPACE_IN_LINK_URLS = new DataKey<>("SPACE_IN_LINK_URLS", false); final public static DataKey SPACE_IN_LINK_ELEMENTS = new DataKey<>("SPACE_IN_LINK_ELEMENTS", false); final public static DataKey WWW_AUTO_LINK_ELEMENT = new DataKey<>("WWW_AUTO_LINK_ELEMENT", false); + final public static DataKey LINK_TEXT_PRIORITY_OVER_LINK_REF = new DataKey<>("LINK_TEXT_PRIORITY_OVER_LINK_REF", false); // if true then link text containing link ref is treated as link text with link ref as text, else link is ignored and link ref is used final public static DataKey REFERENCE_PARAGRAPH_PRE_PROCESSOR = new DataKey<>("REFERENCE_BLOCK_PRE_PROCESSOR", true); final public static DataKey THEMATIC_BREAK_PARSER = new DataKey<>("THEMATIC_BREAK_PARSER", true); @@ -206,6 +207,11 @@ public class Parser implements IParse { */ final public static DataKey HTML_BLOCK_DEEP_PARSE_INDENTED_CODE_INTERRUPTS = new DataKey<>("HTML_BLOCK_DEEP_PARSE_INDENTED_CODE_INTERRUPTS", false); + /** + * Name spaces are allowed in HTML elements, default false for backward compatibility + */ + final public static DataKey HTML_ALLOW_NAME_SPACE = new DataKey<>("HTML_ALLOW_NAME_SPACE", false); + /** * Used by formatter for translation parsing */ @@ -296,7 +302,7 @@ public class Parser implements IParse { final private LinkRefProcessorData linkRefProcessors; final private List inlineParserExtensionFactories; final private InlineParserFactory inlineParserFactory; - final private DataHolder options; + final private @NotNull DataHolder options; Parser(Builder builder) { DataSet options = builder.toImmutable(); @@ -407,9 +413,8 @@ private Document postProcess(Document document) { return document; } - @Nullable @Override - public DataHolder getOptions() { + public @NotNull DataHolder getOptions() { return options; } diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/ParserEmulationProfile.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/ParserEmulationProfile.java index be21c5c07..502762045 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/ParserEmulationProfile.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/ParserEmulationProfile.java @@ -381,6 +381,11 @@ public MutableDataHolder setIn(@NotNull MutableDataHolder dataHolder) { .set(HtmlRenderer.HEADER_ID_GENERATOR_TO_DASH_CHARS, " -") .set(HtmlRenderer.HEADER_ID_GENERATOR_NON_DASH_CHARS, "_") .set(HtmlRenderer.HEADER_ID_GENERATOR_NON_ASCII_TO_LOWERCASE, false) + + // GitHub does not trim trailing spaces of Ref Links in Headings for ID generation + .set(HtmlRenderer.HEADER_ID_REF_TEXT_TRIM_TRAILING_SPACES, false) + // GitHub adds emoji shortcut text to heading id + .set(HtmlRenderer.HEADER_ID_ADD_EMOJI_SHORTCUT, true) ; } else if (this == MULTI_MARKDOWN) { getOptions(dataHolder).setIn(dataHolder); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/HtmlBlockParser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/HtmlBlockParser.java index 2e5152a53..4cedbfbcd 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/HtmlBlockParser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/HtmlBlockParser.java @@ -76,7 +76,7 @@ public Patterns(Parsing parsing, DataHolder options) { Pattern.compile("\\]\\]>") }, { - Pattern.compile("^]|$)", Pattern.CASE_INSENSITIVE), + Pattern.compile("^]|$)", Pattern.CASE_INSENSITIVE), null // terminated by blank line }, { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ListBlockParser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ListBlockParser.java index 778e4d556..32dc553e0 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ListBlockParser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ListBlockParser.java @@ -563,7 +563,7 @@ public BlockStart tryStart(ParserState state, MatchedBlockParser matchedBlockPar } } else if (emulationFamily == FIXED_INDENT) { int currentIndent = state.getIndent(); - if (currentIndent >= myOptions.getCodeIndent()) { + if (currentIndent >= myOptions.getItemIndent()) { return BlockStart.none(); } } else if (emulationFamily == KRAMDOWN) { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ParagraphParser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ParagraphParser.java index 8c5061dd3..73bec56df 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ParagraphParser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/core/ParagraphParser.java @@ -5,6 +5,7 @@ import com.vladsch.flexmark.parser.block.*; import com.vladsch.flexmark.util.ast.BlockContent; import com.vladsch.flexmark.util.sequence.BasedSequence; +import com.vladsch.flexmark.util.sequence.PrefixedSubSequence; public class ParagraphParser extends AbstractBlockParser { @@ -34,7 +35,12 @@ public BlockContinue tryContinue(ParserState state) { @Override public void addLine(ParserState state, BasedSequence line) { - content.add(line, state.getIndent()); + int indent = state.getIndent(); + if (indent > 0) { + content.add(PrefixedSubSequence.repeatOf(' ', indent, line), indent); + } else { + content.add(line, indent); + } } @Override diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/DocumentParser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/DocumentParser.java index 481b8bf37..c4a539ecf 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/DocumentParser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/DocumentParser.java @@ -328,7 +328,6 @@ public Document parse(Reader input) throws IOException { // issue #91 if (charsRead < 0) break; file.append(buffer, 0, charsRead); - if (charsRead < buffer.length) break; } CharSequence source = BasedSequence.of(file.toString()); diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/HtmlDeepParser.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/HtmlDeepParser.java index d9f89e8d8..c270d4a8f 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/HtmlDeepParser.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/HtmlDeepParser.java @@ -1,5 +1,7 @@ package com.vladsch.flexmark.parser.internal; +import com.vladsch.flexmark.ast.util.Parsing; + import java.util.*; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -9,8 +11,8 @@ public enum HtmlMatch { NONE(null, null, false), SCRIPT("<(script)(?:\\s|>|$)", "", true), STYLE("<(style)(?:\\s|>|$)", "", true), - OPEN_TAG("<([A-Za-z][A-Za-z0-9-]*)", "<|/>|\\s/>|>", true), - CLOSE_TAG("", null, true), + OPEN_TAG("<((?:" + Parsing.XML_NAMESPACE + ")[A-Za-z][A-Za-z0-9-]*" + ")", "<|/>|\\s/>|>", true), + CLOSE_TAG("", null, true), NON_TAG("<(![A-Z])", ">", false), TEMPLATE("<([?])", "\\?>", false), COMMENT("<(!--)", "-->", false), @@ -81,6 +83,7 @@ public enum HtmlMatch { for (HtmlMatch state : HtmlMatch.values()) { if (state != HtmlMatch.NONE) { if (startPattern.length() != 0) startPattern.append("|"); + if (state.caseInsentive) { startPattern.append("(?i:"); startPattern.append(state.open.pattern()); @@ -88,6 +91,7 @@ public enum HtmlMatch { } else { startPattern.append(state.open.pattern()); } + PATTERN_MAP[index] = state; } index++; @@ -247,6 +251,14 @@ public void parseHtmlChunk(CharSequence html, boolean blockTagsOnly, boolean par String group = matcher.group(i).toLowerCase(); HtmlMatch htmlMatch = PATTERN_MAP[i]; + + int pos = group.indexOf(':'); + + if (pos >= 0) { + // strip out the namespace + group = group.substring(pos + 1); + } + boolean isBlockTag = myBlockTags.contains(group); if ((blockTagsOnly || !parseNonBlock) && matcher.start() > 0) { diff --git a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/InlineParserImpl.java b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/InlineParserImpl.java index 79623b7cc..888a98c93 100644 --- a/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/InlineParserImpl.java +++ b/flexmark/src/main/java/com/vladsch/flexmark/parser/internal/InlineParserImpl.java @@ -24,6 +24,7 @@ import org.jetbrains.annotations.Nullable; import java.util.*; +import java.util.function.Function; import java.util.regex.Matcher; public class InlineParserImpl extends LightInlineParserImpl implements InlineParser, ParagraphPreProcessor { @@ -924,7 +925,7 @@ protected boolean parseCloseBracket() { String normalizedLabel = Escaping.normalizeReferenceChars(ref, true); if (referenceRepository.containsKey(normalizedLabel)) { BasedSequence sequence = input.subSequence(opener.getStartIndex(), startIndex); - boolean containsLinks = containsLinkRefs(refIsBare ? ref : sequence, opener.getNode().getNext(), true); + boolean containsLinks = containsLinkRefs(refIsBare ? ref : sequence, opener.getNode().getNext(), false); isLinkOrImage = !containsLinks; refIsDefined = true; } else { @@ -1030,6 +1031,7 @@ protected boolean parseCloseBracket() { } } } + appendNode(insertNode); if (insertNode instanceof RefNode) { @@ -1046,7 +1048,7 @@ protected boolean parseCloseBracket() { } insertNode.setCharsFromContent(); } else if (insertNode instanceof InlineLinkNode) { - // set dest and title + // set url and title InlineLinkNode inlineLinkNode = (InlineLinkNode) insertNode; inlineLinkNode.setUrlChars(dest); inlineLinkNode.setTitleChars(title); @@ -1081,15 +1083,25 @@ protected boolean parseCloseBracket() { bracket = bracket.getPrevious(); } - // collapse any link refs contained in this link, they are duds, link takes precedence - // TODO: add a test to see if all link refs should be collapsed or just undefined ones - collapseLinkRefChildren(insertNode, null, true); + if (options.linkTextPriorityOverLinkRef || !containsLinkRefs(insertNode, false)) { + // collapse any link refs contained in this link, they are duds, link takes precedence + collapseLinkRefChildren(insertNode, child -> child instanceof LinkRendered || child.isTentative(), true); + toRemove.unlink(); + } else { + // if contains link ref then treat this as plain text + // leave opener, add children as parent nodes and append closer + insertNode.unlink(); + block.takeChildren(insertNode); + appendText(insertNode.baseSubSequence(((Link) insertNode).getTextClosingMarker().getStartOffset(), insertNode.getEndOffset())); + } } else if (insertNode instanceof RefNode) { // have a link ref, collapse to text any tentative ones contained in it, they are duds - collapseLinkRefChildren(insertNode, true, true); + collapseLinkRefChildren(insertNode, child -> child instanceof LinkRendered || child.isTentative(), true); + toRemove.unlink(); + } else { + toRemove.unlink(); } - toRemove.unlink(); return true; } else { // no link or image index = startIndex; @@ -1099,11 +1111,22 @@ protected boolean parseCloseBracket() { } } - protected static boolean containsLinkRefs(BasedSequence nodeChars, Node next, Boolean isDefined) { + protected static boolean containsLinkRefs(Node node, Boolean isTentative) { + Node next = node.getFirstChild(); + while (next != null) { + if (next instanceof LinkRendered && (isTentative == null || ((LinkRendered) next).isTentative() == isTentative)) { + return true; + } + next = next.getNext(); + } + return false; + } + + protected static boolean containsLinkRefs(BasedSequence nodeChars, Node next, Boolean isTentative) { int startOffset = nodeChars.getStartOffset(); int endOffset = nodeChars.getEndOffset(); while (next != null) { - if (next instanceof LinkRef && (isDefined == null || ((LinkRef) next).isDefined() == isDefined) && !(next.getStartOffset() >= endOffset || next.getEndOffset() <= startOffset)) { + if (next instanceof LinkRefDerived && (isTentative == null || ((LinkRefDerived) next).isTentative() == isTentative) && !(next.getStartOffset() >= endOffset || next.getEndOffset() <= startOffset)) { return true; } next = next.getNext(); @@ -1111,12 +1134,12 @@ protected static boolean containsLinkRefs(BasedSequence nodeChars, Node next, Bo return false; } - protected static void collapseLinkRefChildren(Node node, Boolean isTentative, boolean trimFirstLastChild) { + protected static void collapseLinkRefChildren(Node node, Function isTentative, boolean trimFirstLastChild) { Node child = node.getFirstChild(); boolean hadCollapse = false; while (child != null) { Node nextChild = child.getNext(); - if (child instanceof LinkRefDerived && (isTentative == null || isTentative == ((RefNode) child).isTentative())) { + if (child instanceof LinkRefDerived && (isTentative == null || isTentative.apply((LinkRefDerived) child))) { // need to collapse this one, moving its text contents to text collapseLinkRefChildren(child, isTentative, false); child.unlink(); @@ -1216,12 +1239,8 @@ public BasedSequence parseLinkDestination() { @Override public BasedSequence parseLinkTitle() { BasedSequence title = match(myParsing.LINK_TITLE); - if (title != null) { - // chop off quotes from title and unescape: - return title; //Escaping.unescapeString(title.substring(1, title.length() - 1)); - } else { - return null; - } + // chop off quotes from title and unescape: + return title; //Escaping.unescapeString(title.substring(1, title.length() - 1)); } /** diff --git a/lib/annotations-18.0.0.jar b/lib/annotations-18.0.0.jar deleted file mode 100644 index cb85bb966..000000000 Binary files a/lib/annotations-18.0.0.jar and /dev/null differ diff --git a/lib/commons-codec-1.12-sources.jar b/lib/commons-codec-1.12-sources.jar deleted file mode 100644 index b02e837b9..000000000 Binary files a/lib/commons-codec-1.12-sources.jar and /dev/null differ diff --git a/lib/commons-codec-1.12.jar b/lib/commons-codec-1.12.jar deleted file mode 100644 index 531478ea9..000000000 Binary files a/lib/commons-codec-1.12.jar and /dev/null differ diff --git a/lib/commons-compress-1.18-sources.jar b/lib/commons-compress-1.18-sources.jar deleted file mode 100644 index dfff0e638..000000000 Binary files a/lib/commons-compress-1.18-sources.jar and /dev/null differ diff --git a/lib/commons-compress-1.18.jar b/lib/commons-compress-1.18.jar deleted file mode 100644 index e401046b5..000000000 Binary files a/lib/commons-compress-1.18.jar and /dev/null differ diff --git a/lib/commons-io-2.6-sources.jar b/lib/commons-io-2.6-sources.jar deleted file mode 100644 index 231cbe45d..000000000 Binary files a/lib/commons-io-2.6-sources.jar and /dev/null differ diff --git a/lib/commons-io-2.6.jar b/lib/commons-io-2.6.jar deleted file mode 100644 index 00556b119..000000000 Binary files a/lib/commons-io-2.6.jar and /dev/null differ diff --git a/lib/commons-logging-1.2.jar b/lib/commons-logging-1.2.jar deleted file mode 100644 index 93a3b9f6d..000000000 Binary files a/lib/commons-logging-1.2.jar and /dev/null differ diff --git a/lib/docx4j-JAXB-Internal-8.1.2-sources.jar b/lib/docx4j-JAXB-Internal-8.1.2-sources.jar deleted file mode 100644 index 61df63d2e..000000000 Binary files a/lib/docx4j-JAXB-Internal-8.1.2-sources.jar and /dev/null differ diff --git a/lib/docx4j-JAXB-Internal-8.1.2.jar b/lib/docx4j-JAXB-Internal-8.1.2.jar deleted file mode 100644 index a0f4009a6..000000000 Binary files a/lib/docx4j-JAXB-Internal-8.1.2.jar and /dev/null differ diff --git a/lib/docx4j-core-8.1.2-sources.jar b/lib/docx4j-core-8.1.2-sources.jar deleted file mode 100644 index 76910c67f..000000000 Binary files a/lib/docx4j-core-8.1.2-sources.jar and /dev/null differ diff --git a/lib/docx4j-core-8.1.2.jar b/lib/docx4j-core-8.1.2.jar deleted file mode 100644 index 8d8e41c54..000000000 Binary files a/lib/docx4j-core-8.1.2.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-8.1.2-sources.jar b/lib/docx4j-openxml-objects-8.1.2-sources.jar deleted file mode 100644 index e50258139..000000000 Binary files a/lib/docx4j-openxml-objects-8.1.2-sources.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-8.1.2.jar b/lib/docx4j-openxml-objects-8.1.2.jar deleted file mode 100644 index 9d54ec879..000000000 Binary files a/lib/docx4j-openxml-objects-8.1.2.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-pml-8.1.2-sources.jar b/lib/docx4j-openxml-objects-pml-8.1.2-sources.jar deleted file mode 100644 index b52547b1c..000000000 Binary files a/lib/docx4j-openxml-objects-pml-8.1.2-sources.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-pml-8.1.2.jar b/lib/docx4j-openxml-objects-pml-8.1.2.jar deleted file mode 100644 index 64337ff43..000000000 Binary files a/lib/docx4j-openxml-objects-pml-8.1.2.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-sml-8.1.2-sources.jar b/lib/docx4j-openxml-objects-sml-8.1.2-sources.jar deleted file mode 100644 index 58a077dd4..000000000 Binary files a/lib/docx4j-openxml-objects-sml-8.1.2-sources.jar and /dev/null differ diff --git a/lib/docx4j-openxml-objects-sml-8.1.2.jar b/lib/docx4j-openxml-objects-sml-8.1.2.jar deleted file mode 100644 index 8ec91828a..000000000 Binary files a/lib/docx4j-openxml-objects-sml-8.1.2.jar and /dev/null differ diff --git a/lib/error_prone_annotations-2.3.3-sources.jar b/lib/error_prone_annotations-2.3.3-sources.jar deleted file mode 100644 index 7eebe21da..000000000 Binary files a/lib/error_prone_annotations-2.3.3-sources.jar and /dev/null differ diff --git a/lib/error_prone_annotations-2.3.3.jar b/lib/error_prone_annotations-2.3.3.jar deleted file mode 100644 index 8fe4938d9..000000000 Binary files a/lib/error_prone_annotations-2.3.3.jar and /dev/null differ diff --git a/lib/failureaccess-1.0.1-sources.jar b/lib/failureaccess-1.0.1-sources.jar deleted file mode 100644 index 0c429f237..000000000 Binary files a/lib/failureaccess-1.0.1-sources.jar and /dev/null differ diff --git a/lib/failureaccess-1.0.1.jar b/lib/failureaccess-1.0.1.jar deleted file mode 100644 index 9b56dc751..000000000 Binary files a/lib/failureaccess-1.0.1.jar and /dev/null differ diff --git a/lib/fontbox-2.0.16.jar b/lib/fontbox-2.0.16.jar deleted file mode 100644 index b1ec49b01..000000000 Binary files a/lib/fontbox-2.0.16.jar and /dev/null differ diff --git a/lib/graphics2d-0.24.jar b/lib/graphics2d-0.24.jar deleted file mode 100644 index 7b7c80b19..000000000 Binary files a/lib/graphics2d-0.24.jar and /dev/null differ diff --git a/lib/httpclient-4.5.8-sources.jar b/lib/httpclient-4.5.8-sources.jar deleted file mode 100644 index f4a53df08..000000000 Binary files a/lib/httpclient-4.5.8-sources.jar and /dev/null differ diff --git a/lib/httpclient-4.5.8.jar b/lib/httpclient-4.5.8.jar deleted file mode 100644 index 1e0aa194c..000000000 Binary files a/lib/httpclient-4.5.8.jar and /dev/null differ diff --git a/lib/httpcore-4.4.11-sources.jar b/lib/httpcore-4.4.11-sources.jar deleted file mode 100644 index 53ea01b2d..000000000 Binary files a/lib/httpcore-4.4.11-sources.jar and /dev/null differ diff --git a/lib/httpcore-4.4.11.jar b/lib/httpcore-4.4.11.jar deleted file mode 100644 index c31d40143..000000000 Binary files a/lib/httpcore-4.4.11.jar and /dev/null differ diff --git a/lib/icu4j-59.1.jar b/lib/icu4j-59.1.jar deleted file mode 100644 index 3dc69c8a1..000000000 Binary files a/lib/icu4j-59.1.jar and /dev/null differ diff --git a/lib/jackson-annotations-2.9.0-sources.jar b/lib/jackson-annotations-2.9.0-sources.jar deleted file mode 100644 index dfd62d00c..000000000 Binary files a/lib/jackson-annotations-2.9.0-sources.jar and /dev/null differ diff --git a/lib/jackson-annotations-2.9.0.jar b/lib/jackson-annotations-2.9.0.jar deleted file mode 100644 index c602d75d4..000000000 Binary files a/lib/jackson-annotations-2.9.0.jar and /dev/null differ diff --git a/lib/jackson-core-2.9.8-sources.jar b/lib/jackson-core-2.9.8-sources.jar deleted file mode 100644 index 61edefa8c..000000000 Binary files a/lib/jackson-core-2.9.8-sources.jar and /dev/null differ diff --git a/lib/jackson-core-2.9.8.jar b/lib/jackson-core-2.9.8.jar deleted file mode 100644 index 362f1f393..000000000 Binary files a/lib/jackson-core-2.9.8.jar and /dev/null differ diff --git a/lib/jackson-databind-2.9.8-sources.jar b/lib/jackson-databind-2.9.8-sources.jar deleted file mode 100644 index 85895899f..000000000 Binary files a/lib/jackson-databind-2.9.8-sources.jar and /dev/null differ diff --git a/lib/jackson-databind-2.9.8.jar b/lib/jackson-databind-2.9.8.jar deleted file mode 100644 index 2d8687b5d..000000000 Binary files a/lib/jackson-databind-2.9.8.jar and /dev/null differ diff --git a/lib/jsoup-1.11.3.jar b/lib/jsoup-1.11.3.jar deleted file mode 100644 index 962e35572..000000000 Binary files a/lib/jsoup-1.11.3.jar and /dev/null differ diff --git a/lib/log4j-1.2.17.jar b/lib/log4j-1.2.17.jar deleted file mode 100644 index 1d425cf7d..000000000 Binary files a/lib/log4j-1.2.17.jar and /dev/null differ diff --git a/lib/openhtmltopdf-core-1.0.0.jar b/lib/openhtmltopdf-core-1.0.0.jar deleted file mode 100644 index 5db66fa2e..000000000 Binary files a/lib/openhtmltopdf-core-1.0.0.jar and /dev/null differ diff --git a/lib/openhtmltopdf-jsoup-dom-converter-1.0.0.jar b/lib/openhtmltopdf-jsoup-dom-converter-1.0.0.jar deleted file mode 100644 index ae7cfda28..000000000 Binary files a/lib/openhtmltopdf-jsoup-dom-converter-1.0.0.jar and /dev/null differ diff --git a/lib/openhtmltopdf-pdfbox-1.0.0.jar b/lib/openhtmltopdf-pdfbox-1.0.0.jar deleted file mode 100644 index 2f49509af..000000000 Binary files a/lib/openhtmltopdf-pdfbox-1.0.0.jar and /dev/null differ diff --git a/lib/openhtmltopdf-rtl-support-1.0.0.jar b/lib/openhtmltopdf-rtl-support-1.0.0.jar deleted file mode 100644 index f49eb328f..000000000 Binary files a/lib/openhtmltopdf-rtl-support-1.0.0.jar and /dev/null differ diff --git a/lib/pdfbox-2.0.16.jar b/lib/pdfbox-2.0.16.jar deleted file mode 100644 index 300bb045a..000000000 Binary files a/lib/pdfbox-2.0.16.jar and /dev/null differ diff --git a/lib/slf4j-api-1.7.26-sources.jar b/lib/slf4j-api-1.7.26-sources.jar deleted file mode 100644 index 4ad41410e..000000000 Binary files a/lib/slf4j-api-1.7.26-sources.jar and /dev/null differ diff --git a/lib/slf4j-api-1.7.26.jar b/lib/slf4j-api-1.7.26.jar deleted file mode 100644 index d2f27acea..000000000 Binary files a/lib/slf4j-api-1.7.26.jar and /dev/null differ diff --git a/lib/xmlgraphics-commons-2.3-sources.jar b/lib/xmlgraphics-commons-2.3-sources.jar deleted file mode 100644 index 9a8fe8f08..000000000 Binary files a/lib/xmlgraphics-commons-2.3-sources.jar and /dev/null differ diff --git a/lib/xmlgraphics-commons-2.3.jar b/lib/xmlgraphics-commons-2.3.jar deleted file mode 100644 index 2ae770f5b..000000000 Binary files a/lib/xmlgraphics-commons-2.3.jar and /dev/null differ diff --git a/lib/xmpbox-2.0.16.jar b/lib/xmpbox-2.0.16.jar deleted file mode 100644 index e37902bfa..000000000 Binary files a/lib/xmpbox-2.0.16.jar and /dev/null differ diff --git a/pom.xml b/pom.xml index a7ac72ada..3a2a62832 100644 --- a/pom.xml +++ b/pom.xml @@ -13,7 +13,7 @@ com.vladsch.flexmark flexmark-java - 0.61.0 + 0.64.8 flexmark-java Java re-implementation of commonmark-java based parser, with AST reflecting source elements, full source position tracking, greater parser extensibility. @@ -23,9 +23,7 @@ flexmark - flexmark-all - flexmark-osgi - flexmark-docx-converter + flexmark-core-test flexmark-ext-abbreviation flexmark-ext-admonition flexmark-ext-anchorlink @@ -47,6 +45,7 @@ flexmark-ext-jekyll-tag flexmark-ext-macros flexmark-ext-media-tags + flexmark-ext-resizable-image flexmark-ext-spec-example flexmark-ext-superscript flexmark-ext-tables @@ -57,13 +56,6 @@ flexmark-ext-yaml-front-matter flexmark-ext-youtube-embedded flexmark-ext-zzzzzz - flexmark-core-test - flexmark-html2md-converter - flexmark-integration-test - flexmark-jira-converter - flexmark-pdf-converter - flexmark-profile-pegdown - flexmark-test-specs flexmark-test-util flexmark-tree-iteration flexmark-util @@ -80,10 +72,20 @@ flexmark-util-options flexmark-util-visitor flexmark-youtrack-converter + flexmark-html2md-converter + flexmark-integration-test + flexmark-jira-converter + flexmark-pdf-converter + flexmark-profile-pegdown + flexmark-test-specs + flexmark-all + flexmark-osgi + flexmark-docx-converter UTF-8 + 1684437041 ${project.basedir}/target/apidocs/ false @@ -101,35 +103,41 @@ - - - org.codehaus.mojo - versions-maven-plugin - 2.3 - - false - - - - org.apache.maven.plugins - maven-compiler-plugin - 3.7.0 - - 8 - 8 - - - - org.apache.maven.plugins - maven-assembly-plugin - 2.6 - - - flexmark-all/src/assembly/bin.xml - - - - + + + + org.codehaus.mojo + versions-maven-plugin + 2.11.0 + + false + + + + org.apache.maven.plugins + maven-jar-plugin + 3.2.2 + + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + 11 + + + + org.apache.maven.plugins + maven-assembly-plugin + 3.3.0 + + + flexmark-all/src/assembly/bin.xml + + + + + @@ -170,7 +178,7 @@ org.apache.maven.plugins maven-source-plugin - 3.0.1 + 3.2.1 package-sources @@ -184,7 +192,7 @@ org.apache.maven.plugins maven-javadoc-plugin - 2.10.4 + 3.2.0 package-javadoc @@ -195,7 +203,8 @@ - + + *.internal,*.internal.* false @@ -219,6 +228,18 @@ + + biz.aQute.bnd + bnd-maven-plugin + 6.3.1 + + + + bnd-process + + + + @@ -342,6 +363,11 @@ flexmark-ext-media-tags ${project.version} + + com.vladsch.flexmark + flexmark-ext-resizable-image + ${project.version} + com.vladsch.flexmark flexmark-ext-xwiki-macros @@ -512,7 +538,7 @@ junit junit - 4.12 + 4.13.2 org.openjdk.jmh @@ -538,9 +564,9 @@ Vladimir Schneider - vladimir@vladsch.com + vladimir.schneider@gmail.com vladsch.com - https://vladsch.com/ + https://github.com/vsch/ @@ -550,5 +576,4 @@ https://github.com/vsch/flexmark-java HEAD -