GLib will automatically convert command line options to UTF-8 provided that setlocale(LC_TYPE, "") is called first, and the argument type is STRING (but not FILENAME). Update the CLI tools to take advantage of this behavior, and likewise implement it in fakeglib.
GLib does not automatically convert non-option arguments (i.e. everything remaining in argv after option processing), so manually call g_locale_to_utf8 on these arguments when they represent table names. This should fix the CLI tools when processing non-ASCII table names in non-UTF-8 locales. Also update fakeglib to implement a fast and loose version of g_locale_to_utf8, and factor out some of the code page => iconv name logic in iconv.c so it can be used in our fake g_locale_to_utf8. This adds a new symbol mdb_iconv_name_from_code_page that is not advertised in the main header file. I did not want to include mdbtools.h from fakeglib.c, but maybe that's not important.
Other programs (e.g. gmdb2) use mdb_print_col, so restore the old enum
names and values. MDB_EXPORT_ESCAPE_INVISIBLE can be OR'ed into the
last argument to enable C-style escaping of text fields.
According to the HACKING file, the file's default language ID is stored
in the database header. Use this value instead of a generic English
language locale for indexing JET4 files.
Columns can have their own text sorting rules, including language ID
distinct from the file's language ID, but this is not addressed as we'd
have to break the mdb_index_hash_text function signature, which I'm not
prepared to do just yet.
There appear to be two bytes after the language ID that may indicate
additional sorting flags. These bytes need additional research.
Using the notes and RC4 key provided in the HACKING file, decrypt the
database definition page all at once instead of decrypting individual
fields with ad-hoc keys. Use the newly decrypted header to access the
database code page at offset 0x3C, and use this numeric value to
initialize the iconv converter with an appropriate charset name for
popular windows code pages. More encodings can be added later, with
the eventual goal of getting rid of the MDB_JET3_CHARSET environment
variable.
Note that individual columns can have their own code pages but this
issue is not addressed.
An extra field is added to the MdbFile structure - because this
struct is allocated internally, this should not break the public
ABI.
Finally, only set the db_passwd field if it's a JET3 database (see #144)
Replace the jerry-built UTF-16 => Latin-1 code path with a cross-platform wcstombs solution that emits UTF-8.
This adds an element to the end of the MdbHandle struct, but should not break any existing code.
A run-time option could be added later to emit other encodings, but people who care about such things can just use the iconv code path.
Merge in pull request #108 with a few changes:
* Use the newer mdb_print_col function
* Redefine the last argument of mdb_print_col to be a flags argument
* Rename and redefine the BINEXPORT enums. While technically public,
these were never intended as a public API.
* Name the command line option --escape-c
mdb_init() and mdb_exit() have done nothing for a while.
mdb_get_coltype_string() and mdb_coltype_takes_length() were previously
removed, but remained in the header file by accident.
* Separate -D (date only) and -T (date/time) format options in mdb-export and mdb-json
* New public mdb_set_shortdate_fmt() function in libmdb
* New private(ish) mdb_col_is_shortdate() function
I'm calling it "shortdate" in order to preserve the existing API.
See https://github.com/mdbtools/mdbtools/issues/12
This should fix long-standing complaints about the default bind size
without causing undue memory inflation in existing applications.
Could make this adjustable on the command line later.
Supersedes:
https://github.com/mdbtools/mdbtools/pull/137
Quickstart (requires Clang 6 or later):
$ export LIB_FUZZING_ENGINE=/path/to/fuzzing/library.a
$ ./configure --enable-fuzz-testing
$ make
$ cd src/fuzz
$ make fuzz_mdb
$ ./fuzz_mdb
Also add a new `mdb_open_buffer function` to facilitate in-memory
fuzz-testing. This requires fmemopen, which may not be present on all
systems. The internal API has been reworked to use file streams instead
of file descriptors. This allows reading from memory and reading from
files using a consistent API.
* Replace backend_is_init (formerly is_init) with relationships_table
* Remove deprecated functions that used static variables
* Move static "bound" variable to MdbHandle as relationships_values
Attempt to make the backend handling logic thread-safe. This removes the
last MDB_CONSTRUCTOR. Also get rid of some JAVA junk and make the
remaining static variables in backend.c constant. Finally remove some
obsolete fields from MdbFile.
Add a shim implementing half-assed versions of most of the GLib
functions used by MDB Tools. If GLib is detected at compile-time,
use it, otherwise use the shim.
This work is not complete, as the option-parsing code is not yet
implemented - so most of the command-line tools crash.
If a host project has -Wstrict-prototypes enabled, the mdbtools header
results in warnings. Turn on -Wstrict-prototypes for the whole project
so that these never occur again.
Some Access 2010 files use 0x03 as the version number rather than
0x0103. For this reason I have changed the call to mdb_get_int32 to
mdb_get_byte.
In addition, according to the Library of Congress page:
https://www.loc.gov/preservation/digital/formats/fdd/fdd000463.shtml
Access 2016 uses 0x05 as the version number. I have inferred from the
Wikipedia page that Access 2013 likely uses 0x04.
Adds "-B" (--boolean-words) option to mdb-export, which will reconfigure
mdb/data.c to export TRUE/FALSE for boolean values instead of 1/0. The
option is needed to support BOOLEAN fields on PostgreSQL, which will not
implicitly cast bare 1/0 into a BOOLEAN value. Value literals are the
SQL TRUE/FALSE, and _quoted_ words meaning true/false and _quoted_
'1'/'0'. With this flag the SQL TRUE/FALSE values are output, which should
work with several SQL databases.
PostgreSQL Reference:
http://www.postgresql.org/docs/current/static/datatype-boolean.html
This commit adds another binary output conversion option to convert
binary blobs into hexadecimal notation, similar to the output of the
following command:
xxd -p binaryfile.bin | tr -d '\n'
(In other words, a single line string of hexadecimal characters
representing a binary blob)
When exporting SQL INSERT statements for SQLite and PostgreSQL and when
the hexadecimal notation is specified with 'mdb-export -b hex ...',
special consideration is given to ensure that binary blobs are safely
written to the SQLite or PostgreSQL database.
Signed-off-by: David Hicks <david@hicks.id.au>
__attribute__ does not exist in Visual Studio. Therefore replaced wherever it appeared with a macro:
Redefines MDB_DEPRECATED to support Visual Studio
Define a Constructor MACRO so that __attribute__((constructor/destructor)) behavior is achieved in Visual Studio.
Just using generate_table_schema through mdb_print_schema deletes the purpose of a very good tool. generate_table_schemas is a rewrite of generate_table_schema but sends the data to a char* instead of FILE*. There is NO fmemopen() or similar in Visual Studio, so there is NO way to access memory through a FILE* except for first writing to the disk and then reading from the disk in memory.
I cannot suggest how to handle the case when td == 0 for the dates. The databases I work with often have just 00:00:00 in the DateTime column which is not consistent with the rest of the column either, but I have to deal with it somehow.
Leaving void* where char* is needed as a function parameter returns a compilation error in Visual Studio.
