Korean text is not rendered correctly in the "Browse Data" tab

Details for the issue

What did you do?

Viewed a table with a TEXT column, where at least one row contains Korean text (for example, "한국어" meaning "Korean").

What did you expect to see?

Glyphs that look like 한국어, or at worst the "unsupported character" symbol: □

What did you see instead?

Glyphs rendered as white space:

The text is actually present, as I can copy it out of the cell and into another program (like a web browser open to the Github issue tracker for sqlitebrowser 🙂). It's just being rendered as whitespace.

This is worse than just displaying □, since to a casual observer it would appear that the cell has no data and is empty.

Useful extra information

DB4S v3.12.0 [built for x86_64-little_endian-llp64] on Windows 10 (10.0) (winnt/10.0.19041) [x86_64]
using SQLite Version 3.32.2
and Qt 5.12.8

What font are you using for the Browse Data tab? This is set in the preferences. I use Segoe UI, and it shows these characters fine...

Even Arial shows them...

Ah, I didn't realize that was an option in preferences. It looks like mine defaulted to "MS Shell Dlg 2", but changing it to Segoe UI didn't solve the problem:

Neither did Arial: extended screenshot showing a bunch of other languages

Thanks for the extended screenshot. Weird thats the only entry - all the other entries are OK, as are the square 'what do I display here?' characters. As a test, can you insert this query?
insert into <your table> (<fields>) values ('Korean','한국어');

I don't know what your table or fields are, but I'm wondering if that field contains something else that isn't being displayed, but is still corrupting the entry? You could find this out using the command-line SQLite, possibly. But easier to re-slap in the correct entry and see if it displays.
Aah, what shows in the 'edit database cell' pane if switched to binary mode? For the text you gave me (three characters?) I get this:

Each of the three characters takes 3 bytes in UTF-8, which is why you get 9 bytes in that panel. I see the same thing:

This is the SQL that created that table:

CREATE TABLE languages
(
    id INTEGER NOT NULL PRIMARY KEY,
    short_name TEXT NOT NULL,
    long_name TEXT NOT NULL,
    en_name TEXT NOT NULL
)

Ah, just discovered a few more interesting facts:

The text is also incorrectly rendered in the "Execute SQL" tab's editor (probably expected since both probably use the same text rendering code).
Korean text encoded in decomposed Unicode is rendered correctly; it's only composed characters that are misrendered as white space.

Which produces:

koNFC contains the following bytes: ed959ceab5adec96b4
koNFD contains the following bytes: e18492e185a1e186abe18480e185aee186a8e1848be185a5

(If you aren't familiar with Unicode normalization, those are both valid representations of the same text. NFC normalization uses code points that represent the entire glyph, and NFD normalization uses code points that represent individual parts of the characters and combine with each other to produce the same rendered output.)

If we both have the same bytes in the cell, then I still don't understand how mine works, but yours doesn't.
I assume you're running the 64-bit version of DB4S? Just out of curiosity (we used to have a 'feature' where the x86 version had one version of Qt, while the x64 version used another ... I believe they both use the same now...) can you try the 32-bit version? Can you also try the nightly version? That is the same as the 3.12.0 version, but ... well, lets rule out everything! ;)
Nightly available here: https://nightlies.sqlitebrowser.org/latest/
I'd download the ZIP version (https://nightlies.sqlitebrowser.org/latest/DB.Browser.for.SQLite-win32.zip) which can just be extracted, then run (no install needed).
What is your OS locale set to?

Here's a repro with the nightly you linked:

The original issue report was with the 64-bit version. My OS locale is set to en-US:

Thanks for trying and your patience! Appreciate your efforts. Sorry we haven't nailed it yet, but the information and data you've provided is a good step forward.

No problem, happy to help.

One detail I just thought of that might be relevant is that the encoding of the database is set to UTF-8:

I expect that to be the common case, though.

Yup, mine was that tooooooooo.

@zrneely

I've tested this under Windows. Mine is working too.

DB4S v3.12.0 [built for x86_64-little_endian-llp64] on Windows 10 (10.0) (winnt/10.0.18362) [x86_64]
using SQLite Version 3.32.2
and Qt 5.12.8

@zrneely What happens if you choose the "RTL Text" mode? It's using a different widget (Qt) than the one used in "Text" mode (QScintilla). But given that in the table browser it doesn't work either, I guess that mode won't work either.

This is one of those weird behaviours where I'd love to see any kind of system console output generated by DB4S. Unfortunately, that's not really an option on windows. 😦

Kind of suspecting there's something throwing an error when it gets up to that cell, but there's no way to catch the error message to figure out what's wrong. 🤕

@mgrojo

@mgrojo RTL mode has the same behavior :(

@justinclift I don't think there's any error parsing or loading the data for that cell - I can copy/paste out of the cell and I get the right text, which is why I think it's a rendering issue.

Kind of suspecting there's something throwing an error when it gets up to that cell, but there's no way to catch the error message

Can't messages be thrown out to the application log too? Guess they can't be globally trapped, but if debug code could be slapped around, then the nightlies could at least give some pointers as to whats going on...

@justinclift

@justinclift @MKleusberg @chrisjlocke - this situation makes me think that it might be an idea to have a "debug" build for fault finding. Setting the -DCMAKE_BUILD_TYPE="DEBUG" might be useful in this instance.

Is that where it spits stuff out to the console if started via the command-line?

Yup. Other "logging" methods can be setup later, but it might be useful to see if anything gets spat out by the build during execution.

This is one of those weird behaviours where I'd love to see any kind of system console output generated by DB4S. Unfortunately, that's not really an option on windows. frowning

Kind of suspecting there's something throwing an error when it gets up to that cell, but there's no way to catch the error message to figure out what's wrong. face_with_head_bandage

If I remember correct you get to see the console output when starting DB4S from WinDbg. So that might be worth a try, especially since Qt happily prints all sorts of error messages to the console and this seems to be some sort of Qt problem.

Great idea! windbg printed this output after I loaded my test DB, went to the "browse data" tab, and selected the "languages" table from the table dropdown for the first time:

  OpenType support missing for "Arial", script 11
  OpenType support missing for "MS Shell Dlg 2", script 11
  OpenType support missing for "MS UI Gothic", script 11
  OpenType support missing for "SimSun", script 11
  OpenType support missing for "Arial", script 12
  OpenType support missing for "MS Shell Dlg 2", script 12
  OpenType support missing for "MS UI Gothic", script 12
  OpenType support missing for "SimSun", script 12
  OpenType support missing for "Arial", script 13
  OpenType support missing for "MS Shell Dlg 2", script 13
  OpenType support missing for "MS UI Gothic", script 13
  OpenType support missing for "SimSun", script 13
  OpenType support missing for "Arial", script 17
  OpenType support missing for "MS Shell Dlg 2", script 17
  OpenType support missing for "MS UI Gothic", script 17
  OpenType support missing for "SimSun", script 17
  OpenType support missing for "Arial", script 16
  OpenType support missing for "MS Shell Dlg 2", script 16
  OpenType support missing for "MS UI Gothic", script 16
  OpenType support missing for "SimSun", script 16
(8b60.32b8): C++ EH exception - code e06d7363 (first chance)
(8b60.32b8): C++ EH exception - code e06d7363 (first chance)
(8b60.32b8): C++ EH exception - code e06d7363 (first chance)
  OpenType support missing for "Arial", script 14
  OpenType support missing for "MS Shell Dlg 2", script 14
  OpenType support missing for "MS UI Gothic", script 14
  OpenType support missing for "SimSun", script 14
  OpenType support missing for "Arial", script 18
  OpenType support missing for "MS Shell Dlg 2", script 18
  OpenType support missing for "MS UI Gothic", script 18
  OpenType support missing for "SimSun", script 18
  OpenType support missing for "Arial", script 19
  OpenType support missing for "MS Shell Dlg 2", script 19
  OpenType support missing for "MS UI Gothic", script 19
  OpenType support missing for "SimSun", script 19
  OpenType support missing for "Arial", script 15
  OpenType support missing for "MS Shell Dlg 2", script 15
  OpenType support missing for "MS UI Gothic", script 15
  OpenType support missing for "SimSun", script 15
  OpenType support missing for "Arial", script 32
  OpenType support missing for "MS Shell Dlg 2", script 32
  OpenType support missing for "MS UI Gothic", script 32
  OpenType support missing for "SimSun", script 32
  OpenType support missing for "Arial", script 20
  OpenType support missing for "MS Shell Dlg 2", script 20
  OpenType support missing for "MS UI Gothic", script 20
  OpenType support missing for "SimSun", script 20

So it seems there are some exceptions being thrown (and caught? windbg didn't break on them), along with a bunch of font errors. Unfortunately I don't have a lot of experience debugging C++ in windbg (and I also don't have symbols handy).

OpenType support missing for "Arial", script 11

Why a Windows install is missing support for one of it's core fonts is beyond me. But the proliferation of repeating font names in the list above suggests to me that the OS is expecting a very narrow list of font sizes ONLY (e.g. 8, 10, 18, 24, 28...).

@lucydodo

@lucydodo Do you have any idea about this? Maybe you can help @zrneely

Well, I'm not sure in this case, but.. I have had several similar experiences before, include using 'Qt Linguist'.

In the case of me at that time, I don't know if it is a bug in Windows,
but it was a problem because Korean glyph was not included in the font.

If you think it's worth a try:

Press 'Win + I' to open 'Windows Settings'
Navigate to the 'Personalization' -> 'Fonts'
Click 'Download Fonts for All Languages' and 'OK'

Including screencasts or screenshots might be helpful to understand, but I'm outside now.. 😞

Close this issue because no response has been received for one week.
You can re-open this issue at any time by responding again.

Navigation Menu

Repository navigation

Korean text is not rendered correctly in the "Browse Data" tab #2310

Details for the issue

What did you do?

What did you expect to see?

What did you see instead?

Useful extra information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Korean text is not rendered correctly in the "Browse Data" tab #2310

Description

Details for the issue

What did you do?

What did you expect to see?

What did you see instead?

Useful extra information

Activity

chrisjlocke commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

chrisjlocke commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

chrisjlocke commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

chrisjlocke commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

chrisjlocke commented on Jun 30, 2020

mgrojo commented on Jun 30, 2020

justinclift commented on Jun 30, 2020

zrneely commented on Jun 30, 2020

chrisjlocke commented on Jun 30, 2020

scottfurry commented on Jun 30, 2020

chrisjlocke commented on Jul 1, 2020

scottfurry commented on Jul 1, 2020

MKleusberg commented on Jul 9, 2020

zrneely commented on Jul 9, 2020

scottfurry commented on Jul 9, 2020

mgrojo commented on Sep 22, 2020

lucydodo commented on Sep 23, 2020

lucydodo commented on Sep 30, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions