-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Properly encode exported forms as UTF-8 when using the UTF-8 Beta language option #378
Comments
Hi @joyfullservice , sorry for being late on replies! |
@ollypsilon - Thanks for attaching the sample file! Opening this file in Notepad++ confirms that it is encoded as I also confirmed that the exported file has the expected BOM, which is critical to reading the file with the correct encoding. If you run the following command in the immediate window on your sample file, does it return
I then tried running the sanitizing function directly on this file. If you run this from the immediate Window, is the resulting file correctly encoded in UTF-8? (You should see normal text content, not spaces between every letter.)
Both of these functions appear to be running fine on my system, so I might need to you to do some additional testing/debugging on your system to see if we can pinpoint where the encoding is getting messed up. The really neat thing about having this add-in as a Microsoft Access library database is that you can actually debug and step through the add-in code, just as easily as you can in your own VBA project. You can even make changes in the add-in code and test them. (Just keep in mind that changes will not be saved in the add-in when it is loaded as a library. It has to be loaded as the current database to save changes.) If you would like a snippet to export a single form without all the other GUI overhead, you can use the following code. (Paste into any module in the add-in project when it is loaded as a library.) This would allow you to set some break points and step through the export code to observe how the export and sanitize processes work when processing a form object. To load the add-in as a library, open your sample database. (This makes your sample database the current database.) Then use the button on the toolbar to open the VCS add-in. All you need to do is load the form once, this will automatically load the add-in database as a library. At this point you will see the VCS project in the VBE project explorer, alongside your current project. From there you can paste in the following code snippet into one of the modules on the VCS project (such as Again, just keep in mind that any changes you make to the add-in project while it is loaded as a library will be lost when the library is unloaded (typically when closing Microsoft Access). Public Sub TestExport()
With New clsDbForm
With .Parent
Set .DbObject = CurrentProject.AllForms("Form1")
.Export "C:\...\Form1"
End With
End With
End Sub Let me know what you learn! |
Hi @joyfullservice, thx for all your efforts. Unfortunately, sanitizing still is encoded incorrect, I guess related to HasUcs2Bom returns not expected result. |
I think this is the issue!! The Notice in the first section of the Sanitize function, it checks the BOM to see how to read the source file: In the case of an exported form object, it needs to read the file as "Unicode" (UTF-16 LE). Because the Fixing the In the '---------------------------------------------------------------------------------------
' Procedure : HasUtf8Bom
' Author : Adam Waller
' Date : 7/30/2020
' Purpose : Returns true if the file begins with a UTF-8 BOM
'---------------------------------------------------------------------------------------
'
Public Function HasUtf8Bom(strFilePath As String) As Boolean
HasUtf8Bom = FileHasBom(strFilePath, UTF8_BOM)
End Function
'---------------------------------------------------------------------------------------
' Procedure : HasUcs2Bom
' Author : Adam Waller
' Date : 8/1/2020
' Purpose : Returns true if the file begins with
'---------------------------------------------------------------------------------------
'
Public Function HasUcs2Bom(strFilePath As String) As Boolean
HasUcs2Bom = FileHasBom(strFilePath, UCS2_BOM)
End Function
'---------------------------------------------------------------------------------------
' Procedure : FileHasBom
' Author : Adam Waller
' Date : 8/1/2020
' Purpose : Check for the specified BOM by reading the first few bytes in the file.
'---------------------------------------------------------------------------------------
'
Private Function FileHasBom(strFilePath As String, strBom As String) As Boolean
FileHasBom = (strBom = StrConv(GetFileBytes(strFilePath, Len(strBom)), vbUnicode))
End Function Also of note are the constants defined in I used simple constants to represent the byte order marks, thinking that this would be a more readable way to represent these in the source code. The first thing I would check is to make sure that the constants look like what I have pictured in the screenshot above. If they look like something else, then we would need to address that first. Here is an example of what the Ucs2 BOM looks like in a file: If you could do some testing with the I will look forward to hearing what you are able to learn about reading the BOM from a file when running under UTF-8 (beta) mode. |
I had a look how this unicode thing needs to be understand and how it's working, I was puzzled when reading your UTF-8, UTF-16 LE terms 🤷♂️
When using on non UTF-8 beta enabled system it doesn't make a difference to compare string or check hex code: But on UTF-8 beta enabled system it makes a difference: I guess detection of file encoding could be tweaked. I found this article which describes the challenge:
Based on the sample codes there I could create a function to get/estimated file encoding. Or you can develop your own 😉 |
I did some additional testing on my machine using the UTF-8 beta option, and I think this is going to take some more testing/development work to resolve. For example, at present, the export of the code module is not working correctly for |
The system encoding lookup is primarily used for the export and import of code modules. In this context we don't want to use UTF-8 as the system encoding for VBA modules. #378
In order to reliably detect the BOM headers in files, we need to read the actual bytes instead of comparing to a string constant. The constant works fine on most systems, but notably not on those using the UTF-8 (beta) option in Windows 10. #378
Great news! It looks like there were only a couple things I had to resolve in order for the add-in to function in UTF-8 (beta) mode. I have updated the BOM detection functions to read the bytes directly rather than comparing them to string conversions. The string conversions do not work reliably for these characters in UTF-8 mode, probably because they involve invalid code points under UTF-8. I am now able to export and build the add-in in UTF-8 (beta) mode, as well as export and build the sample database provided by @ollypsilon. I will be uploading a new beta release shortly. Version 4.07 should be able to successfully export and build in both Unicode and non-Unicode environments. Let me know how this works! |
@joyfullservice , I was able to export and build my database on both systems, great! I spotted this only on forms where I use some special characters like currency symbols and language specific characters. |
I did some more testing on this last night, and I have bad news and good news. The bad news is that as far as I can tell from testing, the The good news is that I believe I have a way we can work around this. And it might also push us over the edge to make an important (although significant) change to the way form exports work. The built in This gives us some helpful advantages in the long run. First, it separates two distinctly different types of content into different files. To me it makes sense that While we could use an option to give the user a choice between the combined or split file approach, I am kind of leading towards going with the split files for everyone. I think it brings advantages to everyone, even on systems that don't use UTF-8 beta, and makes the source files more intuitive to use and cleaner on the change tracking. (You will be able to see just from the file names whether a change involved the form definition, the VBA code, or both.) @hecon5, @A9G-Data-Droid, @Tanarri, @Indigo744, feel free to chime in if you have any thoughts regarding the proposal to split the form export into two source files, similar to how VBE works with native VBE forms. |
For full transparency, I'm not using this project for my version control, but I've got a fairly advanced custom process that I use. One thing that I find myself frequently doing is using LoadFromText to import a single form or report at a time. Splitting the form into two exported files would break that functionality. That said, I agree that the benefits outweigh the costs. I also agree that forcing everyone to use the same approach is the better decision for the project in the long run. If I were to switch to this project, I would like to see the add-in include support for importing one form or report at a time. With the split files approach, this would need to be something built-in to the add-in. Great job with the project, by the way. I've been watching from the sidelines and I'm eager to see what the next big release looks like. Keep up the great work 👍 . |
Thanks, Mike! I appreciate your perspective and thoughts on this. Great suggestion on the ability to export/import a single object. That would sure come in handy for testing and tweaking. On an issue like this, I often write a few lines of code to export and import a particular test object, but it would be even easier to just select the item in the navigation pane and click a button on the ribbon. 😄 I just might add that in the near future... Thanks for all the tips and content you share on your blog! Your highlighting of twinBASIC and the ability to easily create a COM add-in is what inspired and empowered me to add the ribbon UI to this project. - Probably the single greatest usability enhancement in the history of the project. 😁 👍 |
I'm late to the party here, but I am going to disagree: I think this splitting should be an "Opt-in" feature (even at the likely cost of then forcing a bifurcated development) Reasoning:
In terms of file extensions:
|
Thanks for pinging me @joyfullservice First time I hear about this UTF8 beta-mode 🐱 I do agree with @hecon5 that splitting files have a lot a consequences, especially on established code base where it would lead to a huge commit and "losing" some history (meaning existing code will be in a new file). On the technical side, if you split files (form and VBA), how will this new export/import function look like? You didn't actually explained so I'm making some educated guess:
If that's case, have you considered to keep a single file, but when importing on a machine with UTF8 beta-mode enabled, split the .bas file temporarily in two (Form / VBA module), apply a proper encoding on the VBA module part (UTF16 instead of local CP) and then import the module using the VBE API? Maybe you've already tried it. I didn't read all linked posts so sorry if that's the case 😸 |
@hecon5 - Thanks for the feedback. I will definitely take those points into consideration. @Indigo744 - You are right, I didn't go into a lot of the technical detail on how I plan to do this. For the export, I would export using the The import process involves first importing the form definition file using We do have the option of stitching the two parts back together after export to produce a single (UTF-8) source file, identical to what Access produces through the I recognize that changing the names of the source files for forms (*.bas to *.frm and *.cls) is a major change, and not one that I take lightly. That's one of the reasons that I am reaching out for feedback from the community. At the same time, I also recognize that sometime the right choice is the difficult one, and while change is inconvenient, it yields the best long-term benefit to the project and those it serves. I think this is what @mwolfe02 was alluding to in some of his comments. Version 4 of the add-in seems like an appropriate place for this kind of breaking change. An example of a difficult change in the past is the decision to use UTF-8-BOM for ALL source files. I was not excited about this initially because it meant a change in nearly every source file, and broke the native compatibility of some of the source files. (For example, you could no longer drag and drop a module file into the IDE and have it import without needing to clean up the BOM characters afterwards. You couldn't run At the end of the day, the primary purpose of the add-in is to be able to generate source files that accurately and intuitively define the individual components of the database at a source file level, and to consistently build the database entirely from source files. If you put yourself into the shoes of a database developer that has never seen the source files before, he will likely open up the combined All of that being said, as @Indigo744 pointed out, this is a really big deal to lose the commit history in git when you rename the files. I don't like that, but it simply how git works. I suppose another options would be to keep the Thanks again for the feedback, and please feel free to add any additional factors that would be helpful to consider in this decision on whether to split out the VBA code in the source files. |
There are many older programming languages that should be written in English. I'm afraid VBA is one of them. Historically, programmers around the world write code and comments in English to avoid these sorts of language support issues. I would recommend avoiding extended characters in VBA\MSAccess and if you need full internationalization support then you should use a more modern programming language and database system. The support MSAccess gets from Microsoft is minimal. It may never be fully UTF8 compliant. The patchwork support that currently exists will lead to a long list of edge cases. What you think is working fine today may be plagued by bugs tomorrow. It's fragile. Even if you upgrade your back-end to SQL to get internationalization compliant storage you will still be limited by the strange encoding issues from VBA on your front-end. |
I've been feeling like a dinosaur for clinging to Mercurial, but this appears to be an area where Mercurial is simply better than git. I'm considering moving from my own custom Export to source script to this project, but I use a completely different naming convention. However, Mercurial supports renaming the files even after they've been created. For example, I can use the following commands to convert from my Form naming convention to match this project's:
Doing this preserves all history. So, for me, renaming files is not such a big deal. But I can see where it would be a major issue for git users (i.e., most of the civilized software development world). Glad I'm a dinosaur 😉 |
So does git. It automatically detects renames as long as the file hasn't changed radically. The issue here is splitting a file into two parts. Which one is the "renamed" one? Obviously, we would want to track the one with the source code. To make this as seemless as possible I would keep the extension so that the old filename |
The way I understand it based on the linked article, git uses automatic rename detection. That's convenient when it works, but it will never be as reliable as an explicit rename command. Perhaps git's Another thing that Mercurial supports (I believe) is having two new files both point to the same source file for history purposes. In other words, even splitting forms into two different files Mercurial would be able to maintain the history in both new files. The initial commit for the form layout properties file would simply show that the code had been deleted. The initial commit for the code-behind file would show that the form layout properties had been deleted. Going forward, you could look at either the form layout file or the code-behind file and see the full history back to the original creation of the form in Access. I only go into this in such depth to help explain why I wasn't more concerned about the loss of file history by splitting the form and report objects into separate files. I think it might be worth exploring exactly what the options and workarounds would be for people using git to maintain file history following such a move. Ideally, there would be a script that could be run to maintain the file history similar to what I showed for Mercurial. Unfortunately, it sounds like that may not be possible. |
Note for the future: https://stackoverflow.com/questions/3887736/keep-git-history-when-splitting-a-file |
Looking at using this as a way to split form source code into two files, one with the VBA code, and the other with the form definition. The git commands would give us a way to upgrade an existing database to use the split file approach without losing the file history. #378
Good news! It looks like with some minor modifications, Raymond Chen's approach to splitting files in git works with batches of files so we only have to use a few commits to pull off the upgrade process while preserving the file history in git. Now I just need to finish fleshing this out in the code... |
Fleshed out an initial (untested) implementation of using git commands to split files while preserving history. #378
This form includes UTF-8 characters in the form controls, and extended characters in both the form and the underlying VBA code. This cannot be exported and imported on UTF-8 systems without splitting the VBA code from the form design. After finishing the functionality to split forms into two source files, this will serve as a test form to ensure that we can successfully build this type of form from source. (This happens more commonly in non-English environments, and is something that I would like to support.) #378
This option (on by default) will save the VBA code from forms and reports as a related .cls file. (Still under development.) #378 Also removed the "Strip out Publish Option" from the options form. I have never heard of a case where this needs to be changed, and it frees up space for the new option we are adding without cluttering the form.
commit 90b402f Merge: 43c7133 3f01386 Author: Josef Pötzl <115746022+josef-poetzl@users.noreply.github.com> Date: Thu Nov 23 11:28:37 2023 +0100 Merge branch 'joyfullservice:dev' into dev commit 3f01386 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 21 09:55:40 2023 -0600 Add note about Access 2007 support See joyfullservice#464 commit 2bf3047 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 16:29:16 2023 -0600 Add default git files if dbs in repository root If you use the default options of no special export folder path defined, the project may likely be in the repository root. Add the default .gitignore and .gitattributes files if they are missing. (This would be the default setup for most projects.) commit b32ff4a Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 14:16:49 2023 -0600 Add dates stored as text to testing database One stored as a regular date string, and the other as an ISO8601 string. (Neither should convert when reading or writing to JSON.) joyfullservice#459 commit cb66895 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 13:40:51 2023 -0600 Add saved date property to testing database Verifies that the round trip conversion of saved date properties is working correctly. (The dates are stored as UTC in source files, but converted to local dates when imported into the database properties.) joyfullservice#459 commit 5009bdc Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 13:38:30 2023 -0600 Turn on date ISO conversion before reading index These dates need to be converted to local dates for internal processing. joyfullservice#459 commit b1bf7bd Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 13:29:05 2023 -0600 Turn off date ISO conversion by default This is only used in the index and certain database properties. joyfullservice#459 commit adfaa01 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 11:54:37 2023 -0600 Refactor date conversion for DB Properties Save custom date properties in ISO (UTC) format in source files, without converting other property types like strings that may parse as dates. joyfullservice#459 commit 64a14de Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 10:27:55 2023 -0600 Log warning for UNC path access errors Failing to convert a path to UNC may not prevent the operation from completing, but it should be handled and logged. Fixes joyfullservice#461 commit 8776104 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 10:19:03 2023 -0600 Adjust detection of system tables Switching to just using a bit flag check to solve joyfullservice#462 commit 259a8df Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 17 09:29:24 2023 -0600 Update comment After removing string padding in the previous commit. commit 9b6e1dd Author: Tanarri <Tanarri@users.noreply.github.com> Date: Fri Nov 17 16:09:28 2023 +0100 Allow sort of operationames with leading spaces (joyfullservice#463) If a operationname has a leading space (like " MyOperation" ) the function "SortItemsByTime" fails. Now sorting will success. commit 3ad141c Author: hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 23:04:01 2023 -0500 Addresses joyfullservice#459 (joyfullservice#460) Addresses joyfullservice#459 commit 2a183df Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 15:59:26 2023 -0600 Solve rare edge case with SQL IN clause Just in case a user has an embedded unquoted path in a string, the colon will be treated as a non-spaced boundary character during formatting. (For Microsoft Access SQL only) Fixes joyfullservice#447 commit 1c3bce6 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 15:19:55 2023 -0600 Add SQL formatting query to testing database This query demonstrates that we can properly parse and format expressions that refer to controls. joyfullservice#457 commit 62fe0fd Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 15:10:22 2023 -0600 Add support for ! boundary character This character is used in Microsoft Access in a query when referring directly to a control on a form, and should be treated similar to a period as a separator between elements. joyfullservice#457 commit 4328713 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 15:08:47 2023 -0600 Implement dialect in SQL formatting This was previously only partially implemented. joyfullservice#457 commit b8dfdcc Merge: 84b6f77 500b891 Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 14:15:39 2023 -0600 Merge pull request joyfullservice#458 from hecon5/ISOOptimizations Additional Optimizations for parsing time commit 500b891 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 12:57:37 2023 -0500 Verify consistent naming and byref passing of strings commit 732f4e5 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 12:53:14 2023 -0500 Bump Version commit 0ef2d02 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 12:32:52 2023 -0500 Cache the format types instead of needing to build them every time. commit 1961800 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 12:31:57 2023 -0500 Pass by ref so we don't need to build more memory use. Optimize Offset string building to only do math when it's required and fix whitespace. commit 26bf030 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Wed Nov 15 12:46:13 2023 -0500 Revert the ConvDateUTC and ConvTimeUTC functions to always parse the "Fast" way first and revert otherwise. this allows the optimization to be used everywhere with no code changes. Ensure that millisecond accuracy is kept for otherse using the function. No Speed impact is noted on my end to doing this. commit 84b6f77 Merge: a666c9d 436dd45 Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 10:04:12 2023 -0600 Merge pull request joyfullservice#456 from joyfullservice/SplitLayoutFromVBA Split form/report layout definitions from VBA code commit 436dd45 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 09:32:12 2023 -0600 Adjust version number I am using the minor version number to represent releases from the main branch, and the build number to continuously increment during the development cycle. commit 68e01a7 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 09:25:32 2023 -0600 Split layout from VBA in testing database Separates the VBA code from the layout definition in the source files. (Applying to testing database now, will apply to main project soon.) commit 923a72b Merge: fe35d3b bc23513 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 09:22:10 2023 -0600 Split files while preserving history commit bc23513 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 09:22:04 2023 -0600 Restore original files commit c28f32b Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 15 09:21:59 2023 -0600 Rename as new files commit fe35d3b Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 15:44:30 2023 -0600 Add Split Files utility to ribbon (Advanced Tools) Also added an informational message box when the split is complete. commit b7985e9 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 15:42:19 2023 -0600 Use faster date parsing for date only values commit e18d818 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 15:41:10 2023 -0600 Update error handling Refactored a number of locations to use the new syntax for On Error Resume Next, and added code to clear expected errors. commit aff7be9 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 09:21:33 2023 -0600 Fix copy-paste oversight joyfullservice#354 commit 286679d Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 09:03:32 2023 -0600 Add high-performance wrapper functions Avoids the use of RegEx when it is not necessary to parse a standard date format. joyfullservice#354 commit d5e76f7 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Nov 14 09:02:39 2023 -0600 Add performance timing to ISO date parsing See joyfullservice#354 commit de4602e Author: hecon5 <54177882+hecon5@users.noreply.github.com> Date: Mon Nov 13 16:56:40 2023 -0500 Fixes joyfullservice#354 and Fixes joyfullservice#452 (joyfullservice#454) From @hecon5: Bump version minor number because it's not clear that the index will allow round trip from prior types in all cases; it worked on my machine, but that may not always be the case. The date types for the index are handled natively by modJsonConverter and should import/export correctly regardless of user's date / time zone or date encoding on machines. commit e723123 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 13 15:51:10 2023 -0600 Rename function Git.Installed sounds better than Git.GitInstalled, and will almost always be called in the context of the git class. commit e2049fa Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 13 15:49:49 2023 -0600 Automate splitting forms and reports Adds a link and some code automation to split forms and reports in existing projects to layout and class files. commit 07bc57c Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 13 15:48:48 2023 -0600 Add change hook for options Used for special processing when certain options change. commit ff706a4 Merge: 9577eab 7b8be8f Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 13 15:18:55 2023 -0600 Merge branch 'SplitLayoutFromVBA' of https://github.com/joyfullservice/msaccess-vcs-addin into SplitLayoutFromVBA commit 9577eab Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 13 15:08:11 2023 -0600 Add wiki page for Split Files Describes the process in a little more detail. commit 7b8be8f Author: Tanarri <Tanarri@users.noreply.github.com> Date: Fri Nov 10 15:24:54 2023 +0100 Implement correction according to rubberduck (joyfullservice#453) replace VBA commands: format with format$ trim with trim$ commit 925b019 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 17:04:08 2023 -0600 Check for diff tool before comparing objects commit 4f9247d Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 17:02:50 2023 -0600 Fix bugs in build logic Uncovered these while testing. commit 8f1649a Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 16:18:47 2023 -0600 Code cleanup and minor tweaks commit 9069232 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 15:45:32 2023 -0600 Add support to overlay VBA code after import For some (rare) situations, it is necessary to push the VBA code directly using VBE to preserve certain extended characters that may be corrupted in a regular round-trip export/import cycle. commit 932ac84 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 15:41:33 2023 -0600 Rework merging source content before import Cleaning this up to avoid reading and writing the file additional times while merging content from different sources. (Print settings, VBA code) commit 1390f2c Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 15:37:32 2023 -0600 Move source reading function This is used in several areas, and allows us to maintain the source file encoding determination in a single location. commit 54e070a Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Nov 8 15:35:24 2023 -0600 Move print settings processing to clsSourceParser This keeps the LoadComponentFromText function cleaner and easier to read. commit 43c7133 Merge: a666c9d 3e82751 Author: Josef Pötzl <115746022+josef-poetzl@users.noreply.github.com> Date: Wed Nov 8 01:04:24 2023 +0100 Merge branch 'dev-jp' into dev commit da9f94a Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Nov 6 16:11:33 2023 -0600 Don't auto split layout/VBA for existing projects For existing projects in git repositories, form and report layouts should not be automatically split from the VBA code classes. There is another process that will allow us to split the files while preserving history in both files, but this involves a couple commits and requires a clean branch. For existing projects, this is a manual upgrade (option changes). For new projects, it can happen by default. commit 2d21eb6 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Fri Nov 3 11:48:21 2023 -0500 Verify ribbon active state when the add-in loads Ensure that the ribbon is active when installing or activating the add-in. See joyfullservice#451 commit c2ffccf Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Nov 2 17:27:43 2023 -0500 Refactor for name change commit 47c6506 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Nov 2 17:21:57 2023 -0500 Rename Sanitize class to SourceParser This better reflects the expanded role of the class. commit 89101c5 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Nov 2 15:06:49 2023 -0500 Refactor form/report export to split VBA Export is now splitting the VBA from Form and Report objects to separate files with a .cls extension. Moving on to the code that will stitch these files back together before import. commit c87261a Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Oct 25 16:49:04 2023 -0500 Refactor class variables commit 1c0d3f4 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Oct 25 16:30:30 2023 -0500 Refactor sanitizing to use class Updating the existing code to use the new class. commit 8e70ad2 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Oct 25 16:20:38 2023 -0500 Convert Sanitize module to class In some cases sanitizing a source file actually creates two distinct outputs. A layout file and a code file. Rather than making the sanitize function more complicated with byref outputs and non-obvious side effects, I am taking the approach of a more explicit object-oriented route where the code is easier to understand and maintain. (And also allows for future enhancements such as SQL extraction for query definition files.) commit 1ca6886 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Oct 25 16:08:09 2023 -0500 Adjust indenting (minor change) commit 0beaee5 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Wed Oct 25 11:53:30 2023 -0500 Support "|" in performance log entry names Refactored parsing the key from the performance item so that we are not dependent upon a unique delimiter. The timing value is always a number, so we can be confident that the first pipe character is the delimiter. The text after that can be anything, including pipe characters. joyfullservice#450 commit 07e364e Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Oct 24 14:16:53 2023 -0500 Refactor code module export to shared function This logic will be shared when exporting code modules from forms and reports. commit 40cffbc Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Tue Oct 24 13:49:52 2023 -0500 Add option to SplitLayoutFromVBA This option (on by default) will save the VBA code from forms and reports as a related .cls file. (Still under development.) joyfullservice#378 Also removed the "Strip out Publish Option" from the options form. I have never heard of a case where this needs to be changed, and it frees up space for the new option we are adding without cluttering the form. commit a666c9d Author: hecon5 <54177882+hecon5@users.noreply.github.com> Date: Fri Oct 20 16:33:38 2023 -0400 Fix issue with LogUnhandledErrors and simplify use. (joyfullservice#449) commit 5362b37 Merge: 8f2f325 490d82e Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Fri Oct 20 13:54:57 2023 -0500 Merge pull request joyfullservice#448 from bclothier/MiscBugFixes Misc bug fixes commit 490d82e Author: bclothier <bgclothier@gmail.com> Date: Fri Oct 20 11:47:41 2023 -0500 The AutoClose may run after the form has closed (e.g. if the user is quick to close it) which may result in an error about object members not available. Since the form is closed, there's no point in setting the timer interval. To avoid the error when debugging, we add a IsLoaded check and skip it if it's not loaded. commit 5b5e037 Author: bclothier <bgclothier@gmail.com> Date: Fri Oct 20 11:46:05 2023 -0500 Add more types of queries that should not be formatted by SQL formatter because they are a variant of pass-through queries. commit 736c33c Author: bclothier <bgclothier@gmail.com> Date: Fri Oct 20 11:45:31 2023 -0500 Fix a subscript out of range error where the tokens advance beyond the end of the string but the function GetNextTokenID returns 0, which then fails within FormatSQL function since there is no member at index 0. It's not clear why this only fails every second time a query is exported but it is the case where if it fails, exporting it next time will not yield the error. Do it 3rd time, then it fails. commit 8f2f325 Merge: 6cdac0c 0d9ac0a Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 18:48:28 2023 -0500 Merge pull request joyfullservice#446 from bclothier/MiscBugFixes Misc bug fixes commit 0d9ac0a Author: bclothier <bgclothier@gmail.com> Date: Thu Oct 19 18:19:17 2023 -0500 The export log was littered with bunch of warnings about unclosed blocks. This seems to be due to not closing it when evaluating the UseTheme. Even if we skipped it, we still need to remove it from m_colBlocks to balance everything out. commit 4703b7b Author: bclothier <bgclothier@gmail.com> Date: Thu Oct 19 18:17:33 2023 -0500 Add a check when loading XML and verify it was successfully parsed. This avoid generating a bad export where the data are not actually exported due to invalid XML being generated by Application.ExportXML. Unfortunately, if a table contains any characters that aren't valid for XML document, it won't try to escape them and include them as literals. Even if they were escaped, they might not be accepted anyway. XML specifications forbids having any characters in 0x01-0x31 range so if a table data contains such characters, this can cause the XML export to fail. In this case, tab delimited will have to be used instead. However, the previous version was simply silently exporting as if everything is hunky-dory when it's not. Hence, the error. commit 734d835 Author: bclothier <bgclothier@gmail.com> Date: Thu Oct 19 18:12:02 2023 -0500 The logic for checking of existence of git files wasn't always working as expected due to searching the current directory rather than using the export folder. commit 6cdac0c Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 16:20:29 2023 -0500 Add option to pass path to build API You can now specify the source files path when you initiate a build through the API. This allows automated builds to be run even if a copy of the database does not yet exist. (Such as after checking out a project in an automated CI workflow.) joyfullservice#430 commit b418a54 Merge: 1cda774 b1af656 Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 15:21:12 2023 -0500 Merge pull request joyfullservice#441 from hecon5/clsPerformanceUpdates Updating clsPerformance, as some objects never restart timing, and wh… commit b1af656 Merge: 8c6d7f9 1cda774 Author: Adam Waller <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 15:20:32 2023 -0500 Merge branch 'dev' into clsPerformanceUpdates commit 8c6d7f9 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 15:18:28 2023 -0500 Resolve conflict with upstream file Putting the comma after the argument seems to be the preferred industry-standard approach, based on ChatGPT and Bard. commit eae289b Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Thu Oct 19 15:41:03 2023 -0400 Update based on feedback from @joyfullservice. commit 1cda774 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Thu Oct 19 14:02:29 2023 -0500 Allow wrapping of long names in performance class Extending the performance class to allow the wrapping of long names used for categories or operations. (Not really needed within this project, but could potentially be helpful in the future with translations.) joyfullservice#441 commit 3273089 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Oct 2 11:29:03 2023 -0500 Refine some dialect-specific SQL string quotations Backticks only apply to MySQL, while square brackets are used with MSSQL and Access. joyfullservice#442 commit 62fde81 Author: joyfullservice <joyfullservice@users.noreply.github.com> Date: Mon Oct 2 11:19:38 2023 -0500 Update API examples Removed dependency on an external function, and added an example for building from source. commit 6e8007a Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Sat Sep 23 09:26:30 2023 -0400 Bump Version commit bd24538 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Sat Sep 23 09:21:13 2023 -0400 This isn't actually used. commit 4a27077 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Sat Sep 23 09:19:57 2023 -0400 Fixing Private/Public declarations. commit 53394a8 Author: Hecon5 <54177882+hecon5@users.noreply.github.com> Date: Sat Sep 23 09:03:00 2023 -0400 Updating clsPerformance, as some objects never restart timing, and when resetting some objects are not cleared. Fixes joyfullservice#331
* Updating clsPerformance, as some objects never restart timing, and when resetting some objects are not cleared. Fixes joyfullservice#331 * Fixing Private/Public declarations. * This isn't actually used. * Bump Version * Update API examples Removed dependency on an external function, and added an example for building from source. * Refine some dialect-specific SQL string quotations Backticks only apply to MySQL, while square brackets are used with MSSQL and Access. joyfullservice#442 * Allow wrapping of long names in performance class Extending the performance class to allow the wrapping of long names used for categories or operations. (Not really needed within this project, but could potentially be helpful in the future with translations.) joyfullservice#441 * Update based on feedback from @joyfullservice. * Resolve conflict with upstream file Putting the comma after the argument seems to be the preferred industry-standard approach, based on ChatGPT and Bard. * Add option to pass path to build API You can now specify the source files path when you initiate a build through the API. This allows automated builds to be run even if a copy of the database does not yet exist. (Such as after checking out a project in an automated CI workflow.) joyfullservice#430 * The logic for checking of existence of git files wasn't always working as expected due to searching the current directory rather than using the export folder. * Add a check when loading XML and verify it was successfully parsed. This avoid generating a bad export where the data are not actually exported due to invalid XML being generated by Application.ExportXML. Unfortunately, if a table contains any characters that aren't valid for XML document, it won't try to escape them and include them as literals. Even if they were escaped, they might not be accepted anyway. XML specifications forbids having any characters in 0x01-0x31 range so if a table data contains such characters, this can cause the XML export to fail. In this case, tab delimited will have to be used instead. However, the previous version was simply silently exporting as if everything is hunky-dory when it's not. Hence, the error. * The export log was littered with bunch of warnings about unclosed blocks. This seems to be due to not closing it when evaluating the UseTheme. Even if we skipped it, we still need to remove it from m_colBlocks to balance everything out. * Fix a subscript out of range error where the tokens advance beyond the end of the string but the function GetNextTokenID returns 0, which then fails within FormatSQL function since there is no member at index 0. It's not clear why this only fails every second time a query is exported but it is the case where if it fails, exporting it next time will not yield the error. Do it 3rd time, then it fails. * Add more types of queries that should not be formatted by SQL formatter because they are a variant of pass-through queries. * The AutoClose may run after the form has closed (e.g. if the user is quick to close it) which may result in an error about object members not available. Since the form is closed, there's no point in setting the timer interval. To avoid the error when debugging, we add a IsLoaded check and skip it if it's not loaded. * Fix issue with LogUnhandledErrors and simplify use. (joyfullservice#449) * Add option to SplitLayoutFromVBA This option (on by default) will save the VBA code from forms and reports as a related .cls file. (Still under development.) joyfullservice#378 Also removed the "Strip out Publish Option" from the options form. I have never heard of a case where this needs to be changed, and it frees up space for the new option we are adding without cluttering the form. * Refactor code module export to shared function This logic will be shared when exporting code modules from forms and reports. * Support "|" in performance log entry names Refactored parsing the key from the performance item so that we are not dependent upon a unique delimiter. The timing value is always a number, so we can be confident that the first pipe character is the delimiter. The text after that can be anything, including pipe characters. joyfullservice#450 * Adjust indenting (minor change) * Convert Sanitize module to class In some cases sanitizing a source file actually creates two distinct outputs. A layout file and a code file. Rather than making the sanitize function more complicated with byref outputs and non-obvious side effects, I am taking the approach of a more explicit object-oriented route where the code is easier to understand and maintain. (And also allows for future enhancements such as SQL extraction for query definition files.) * Refactor sanitizing to use class Updating the existing code to use the new class. * Refactor class variables * Refactor form/report export to split VBA Export is now splitting the VBA from Form and Report objects to separate files with a .cls extension. Moving on to the code that will stitch these files back together before import. * Rename Sanitize class to SourceParser This better reflects the expanded role of the class. * Refactor for name change * Verify ribbon active state when the add-in loads Ensure that the ribbon is active when installing or activating the add-in. See joyfullservice#451 * Don't auto split layout/VBA for existing projects For existing projects in git repositories, form and report layouts should not be automatically split from the VBA code classes. There is another process that will allow us to split the files while preserving history in both files, but this involves a couple commits and requires a clean branch. For existing projects, this is a manual upgrade (option changes). For new projects, it can happen by default. * Move print settings processing to clsSourceParser This keeps the LoadComponentFromText function cleaner and easier to read. * Move source reading function This is used in several areas, and allows us to maintain the source file encoding determination in a single location. * Rework merging source content before import Cleaning this up to avoid reading and writing the file additional times while merging content from different sources. (Print settings, VBA code) * Add support to overlay VBA code after import For some (rare) situations, it is necessary to push the VBA code directly using VBE to preserve certain extended characters that may be corrupted in a regular round-trip export/import cycle. * Code cleanup and minor tweaks * Fix bugs in build logic Uncovered these while testing. * Check for diff tool before comparing objects * Implement correction according to rubberduck (joyfullservice#453) replace VBA commands: format with format$ trim with trim$ * Add wiki page for Split Files Describes the process in a little more detail. * Add change hook for options Used for special processing when certain options change. * Automate splitting forms and reports Adds a link and some code automation to split forms and reports in existing projects to layout and class files. * Rename function Git.Installed sounds better than Git.GitInstalled, and will almost always be called in the context of the git class. * Fixes joyfullservice#354 and Fixes joyfullservice#452 (joyfullservice#454) From @hecon5: Bump version minor number because it's not clear that the index will allow round trip from prior types in all cases; it worked on my machine, but that may not always be the case. The date types for the index are handled natively by modJsonConverter and should import/export correctly regardless of user's date / time zone or date encoding on machines. * Add performance timing to ISO date parsing See joyfullservice#354 * Add high-performance wrapper functions Avoids the use of RegEx when it is not necessary to parse a standard date format. joyfullservice#354 * Fix copy-paste oversight joyfullservice#354 * Update error handling Refactored a number of locations to use the new syntax for On Error Resume Next, and added code to clear expected errors. * Use faster date parsing for date only values * Add Split Files utility to ribbon (Advanced Tools) Also added an informational message box when the split is complete. * Rename as new files * Restore original files * Split layout from VBA in testing database Separates the VBA code from the layout definition in the source files. (Applying to testing database now, will apply to main project soon.) * Adjust version number I am using the minor version number to represent releases from the main branch, and the build number to continuously increment during the development cycle. * Revert the ConvDateUTC and ConvTimeUTC functions to always parse the "Fast" way first and revert otherwise. this allows the optimization to be used everywhere with no code changes. Ensure that millisecond accuracy is kept for otherse using the function. No Speed impact is noted on my end to doing this. * Pass by ref so we don't need to build more memory use. Optimize Offset string building to only do math when it's required and fix whitespace. * Cache the format types instead of needing to build them every time. * Bump Version * Verify consistent naming and byref passing of strings * Implement dialect in SQL formatting This was previously only partially implemented. joyfullservice#457 * Add support for ! boundary character This character is used in Microsoft Access in a query when referring directly to a control on a form, and should be treated similar to a period as a separator between elements. joyfullservice#457 * Add SQL formatting query to testing database This query demonstrates that we can properly parse and format expressions that refer to controls. joyfullservice#457 * Solve rare edge case with SQL IN clause Just in case a user has an embedded unquoted path in a string, the colon will be treated as a non-spaced boundary character during formatting. (For Microsoft Access SQL only) Fixes joyfullservice#447 * Addresses joyfullservice#459 (joyfullservice#460) Addresses joyfullservice#459 * Allow sort of operationames with leading spaces (joyfullservice#463) If a operationname has a leading space (like " MyOperation" ) the function "SortItemsByTime" fails. Now sorting will success. * Update comment After removing string padding in the previous commit. * Adjust detection of system tables Switching to just using a bit flag check to solve joyfullservice#462 * Log warning for UNC path access errors Failing to convert a path to UNC may not prevent the operation from completing, but it should be handled and logged. Fixes joyfullservice#461 * Refactor date conversion for DB Properties Save custom date properties in ISO (UTC) format in source files, without converting other property types like strings that may parse as dates. joyfullservice#459 * Turn off date ISO conversion by default This is only used in the index and certain database properties. joyfullservice#459 * Turn on date ISO conversion before reading index These dates need to be converted to local dates for internal processing. joyfullservice#459 * Add saved date property to testing database Verifies that the round trip conversion of saved date properties is working correctly. (The dates are stored as UTC in source files, but converted to local dates when imported into the database properties.) joyfullservice#459 * Add dates stored as text to testing database One stored as a regular date string, and the other as an ISO8601 string. (Neither should convert when reading or writing to JSON.) joyfullservice#459 * Add default git files if dbs in repository root If you use the default options of no special export folder path defined, the project may likely be in the repository root. Add the default .gitignore and .gitattributes files if they are missing. (This would be the default setup for most projects.) * Add note about Access 2007 support See joyfullservice#464 --------- Co-authored-by: Hecon5 <54177882+hecon5@users.noreply.github.com> Co-authored-by: joyfullservice <joyfullservice@users.noreply.github.com> Co-authored-by: bclothier <bgclothier@gmail.com> Co-authored-by: Tanarri <Tanarri@users.noreply.github.com>
* Updating clsPerformance, as some objects never restart timing, and when resetting some objects are not cleared. Fixes joyfullservice#331 * Fixing Private/Public declarations. * This isn't actually used. * Bump Version * Update API examples Removed dependency on an external function, and added an example for building from source. * Refine some dialect-specific SQL string quotations Backticks only apply to MySQL, while square brackets are used with MSSQL and Access. joyfullservice#442 * Allow wrapping of long names in performance class Extending the performance class to allow the wrapping of long names used for categories or operations. (Not really needed within this project, but could potentially be helpful in the future with translations.) joyfullservice#441 * Update based on feedback from @joyfullservice. * Resolve conflict with upstream file Putting the comma after the argument seems to be the preferred industry-standard approach, based on ChatGPT and Bard. * Add option to pass path to build API You can now specify the source files path when you initiate a build through the API. This allows automated builds to be run even if a copy of the database does not yet exist. (Such as after checking out a project in an automated CI workflow.) joyfullservice#430 * The logic for checking of existence of git files wasn't always working as expected due to searching the current directory rather than using the export folder. * Add a check when loading XML and verify it was successfully parsed. This avoid generating a bad export where the data are not actually exported due to invalid XML being generated by Application.ExportXML. Unfortunately, if a table contains any characters that aren't valid for XML document, it won't try to escape them and include them as literals. Even if they were escaped, they might not be accepted anyway. XML specifications forbids having any characters in 0x01-0x31 range so if a table data contains such characters, this can cause the XML export to fail. In this case, tab delimited will have to be used instead. However, the previous version was simply silently exporting as if everything is hunky-dory when it's not. Hence, the error. * The export log was littered with bunch of warnings about unclosed blocks. This seems to be due to not closing it when evaluating the UseTheme. Even if we skipped it, we still need to remove it from m_colBlocks to balance everything out. * Fix a subscript out of range error where the tokens advance beyond the end of the string but the function GetNextTokenID returns 0, which then fails within FormatSQL function since there is no member at index 0. It's not clear why this only fails every second time a query is exported but it is the case where if it fails, exporting it next time will not yield the error. Do it 3rd time, then it fails. * Add more types of queries that should not be formatted by SQL formatter because they are a variant of pass-through queries. * The AutoClose may run after the form has closed (e.g. if the user is quick to close it) which may result in an error about object members not available. Since the form is closed, there's no point in setting the timer interval. To avoid the error when debugging, we add a IsLoaded check and skip it if it's not loaded. * Fix issue with LogUnhandledErrors and simplify use. (joyfullservice#449) * Add option to SplitLayoutFromVBA This option (on by default) will save the VBA code from forms and reports as a related .cls file. (Still under development.) joyfullservice#378 Also removed the "Strip out Publish Option" from the options form. I have never heard of a case where this needs to be changed, and it frees up space for the new option we are adding without cluttering the form. * Refactor code module export to shared function This logic will be shared when exporting code modules from forms and reports. * Support "|" in performance log entry names Refactored parsing the key from the performance item so that we are not dependent upon a unique delimiter. The timing value is always a number, so we can be confident that the first pipe character is the delimiter. The text after that can be anything, including pipe characters. joyfullservice#450 * Adjust indenting (minor change) * Convert Sanitize module to class In some cases sanitizing a source file actually creates two distinct outputs. A layout file and a code file. Rather than making the sanitize function more complicated with byref outputs and non-obvious side effects, I am taking the approach of a more explicit object-oriented route where the code is easier to understand and maintain. (And also allows for future enhancements such as SQL extraction for query definition files.) * Refactor sanitizing to use class Updating the existing code to use the new class. * Refactor class variables * Refactor form/report export to split VBA Export is now splitting the VBA from Form and Report objects to separate files with a .cls extension. Moving on to the code that will stitch these files back together before import. * Rename Sanitize class to SourceParser This better reflects the expanded role of the class. * Refactor for name change * Verify ribbon active state when the add-in loads Ensure that the ribbon is active when installing or activating the add-in. See joyfullservice#451 * Don't auto split layout/VBA for existing projects For existing projects in git repositories, form and report layouts should not be automatically split from the VBA code classes. There is another process that will allow us to split the files while preserving history in both files, but this involves a couple commits and requires a clean branch. For existing projects, this is a manual upgrade (option changes). For new projects, it can happen by default. * Move print settings processing to clsSourceParser This keeps the LoadComponentFromText function cleaner and easier to read. * Move source reading function This is used in several areas, and allows us to maintain the source file encoding determination in a single location. * Rework merging source content before import Cleaning this up to avoid reading and writing the file additional times while merging content from different sources. (Print settings, VBA code) * Add support to overlay VBA code after import For some (rare) situations, it is necessary to push the VBA code directly using VBE to preserve certain extended characters that may be corrupted in a regular round-trip export/import cycle. * Code cleanup and minor tweaks * Fix bugs in build logic Uncovered these while testing. * Check for diff tool before comparing objects * Implement correction according to rubberduck (joyfullservice#453) replace VBA commands: format with format$ trim with trim$ * Add wiki page for Split Files Describes the process in a little more detail. * Add change hook for options Used for special processing when certain options change. * Automate splitting forms and reports Adds a link and some code automation to split forms and reports in existing projects to layout and class files. * Rename function Git.Installed sounds better than Git.GitInstalled, and will almost always be called in the context of the git class. * Fixes joyfullservice#354 and Fixes joyfullservice#452 (joyfullservice#454) From @hecon5: Bump version minor number because it's not clear that the index will allow round trip from prior types in all cases; it worked on my machine, but that may not always be the case. The date types for the index are handled natively by modJsonConverter and should import/export correctly regardless of user's date / time zone or date encoding on machines. * Add performance timing to ISO date parsing See joyfullservice#354 * Add high-performance wrapper functions Avoids the use of RegEx when it is not necessary to parse a standard date format. joyfullservice#354 * Fix copy-paste oversight joyfullservice#354 * Update error handling Refactored a number of locations to use the new syntax for On Error Resume Next, and added code to clear expected errors. * Use faster date parsing for date only values * Add Split Files utility to ribbon (Advanced Tools) Also added an informational message box when the split is complete. * Rename as new files * Restore original files * Split layout from VBA in testing database Separates the VBA code from the layout definition in the source files. (Applying to testing database now, will apply to main project soon.) * Adjust version number I am using the minor version number to represent releases from the main branch, and the build number to continuously increment during the development cycle. * Revert the ConvDateUTC and ConvTimeUTC functions to always parse the "Fast" way first and revert otherwise. this allows the optimization to be used everywhere with no code changes. Ensure that millisecond accuracy is kept for otherse using the function. No Speed impact is noted on my end to doing this. * Pass by ref so we don't need to build more memory use. Optimize Offset string building to only do math when it's required and fix whitespace. * Cache the format types instead of needing to build them every time. * Bump Version * Verify consistent naming and byref passing of strings * Implement dialect in SQL formatting This was previously only partially implemented. joyfullservice#457 * Add support for ! boundary character This character is used in Microsoft Access in a query when referring directly to a control on a form, and should be treated similar to a period as a separator between elements. joyfullservice#457 * Add SQL formatting query to testing database This query demonstrates that we can properly parse and format expressions that refer to controls. joyfullservice#457 * Solve rare edge case with SQL IN clause Just in case a user has an embedded unquoted path in a string, the colon will be treated as a non-spaced boundary character during formatting. (For Microsoft Access SQL only) Fixes joyfullservice#447 * Addresses joyfullservice#459 (joyfullservice#460) Addresses joyfullservice#459 * Allow sort of operationames with leading spaces (joyfullservice#463) If a operationname has a leading space (like " MyOperation" ) the function "SortItemsByTime" fails. Now sorting will success. * Update comment After removing string padding in the previous commit. * Adjust detection of system tables Switching to just using a bit flag check to solve joyfullservice#462 * Log warning for UNC path access errors Failing to convert a path to UNC may not prevent the operation from completing, but it should be handled and logged. Fixes joyfullservice#461 * Refactor date conversion for DB Properties Save custom date properties in ISO (UTC) format in source files, without converting other property types like strings that may parse as dates. joyfullservice#459 * Turn off date ISO conversion by default This is only used in the index and certain database properties. joyfullservice#459 * Turn on date ISO conversion before reading index These dates need to be converted to local dates for internal processing. joyfullservice#459 * Add saved date property to testing database Verifies that the round trip conversion of saved date properties is working correctly. (The dates are stored as UTC in source files, but converted to local dates when imported into the database properties.) joyfullservice#459 * Add dates stored as text to testing database One stored as a regular date string, and the other as an ISO8601 string. (Neither should convert when reading or writing to JSON.) joyfullservice#459 * Add default git files if dbs in repository root If you use the default options of no special export folder path defined, the project may likely be in the repository root. Add the default .gitignore and .gitattributes files if they are missing. (This would be the default setup for most projects.) * Add note about Access 2007 support See joyfullservice#464 * Add test for Public Creatable class instance This is an undocumented property value that is sometimes used in the wild. Currently when you build from source, PublicCreatable (5) classes are converted to PublicNotCreatable (2). This instancing property can be set in VBA, and we want the imported class to match what was exported. This test currently fails, but will pass when the add-in is updated to support this property. * Support for PublicCreatable instancing for classes This (technically undocumented) technique allows class objects to be created by external projects without using factory methods. This approach was used in some of my projects, so it was important for me to see this property correctly set when the application was built from source. * Check VCS version before export Checks the VCS version before export to warn the user if we are running an export with an older version of VCS than was last used on this project. joyfullservice#465 * Check VCS version on build Check the VCS version before building, and warn if the project version is greater than the installed version. joyfullservice#465 * Reset error break mode after loading options The ability to break and debug VBA errors is dependent on an option value that is saved with each project. The on error directive should be reset after loading the project options to ensure that we can successfully break on errors. * Include name prefix with VBA code overlay Testing this on a machine using the Unicode BETA option in Windows. * Resolve build path during options upgrade We may not have a database open when upgrading options. Fall back to the path used to load the project options to determine the source file path. joyfullservice#467 * Add some documentation for merge build Taking some time to document the intended behavior of the merge build functionality as we work through some bugs. joyfullservice#471, joyfullservice#81 * Update Merge-Build.md Expand table of expected behavior. * Remove git integration for getting modified source I don't think this is actually being used in the wild, and it simplifies the process to have only a single code path for detecting changed source files. * Add missing database objects on merge If a source file exists, but no matching database object exists, we should merge the source file into the database. joyfullservice#471 * Fix SQL export of non-formatted queries Pass-through queries are now exported as SQL again. * Refactor source modified date for multiple files Some types of components, such as tables, forms, reports, queries, and shared images may use multiple source files to represent a single database component. We may need to check all of the related source files to accurately determine the latest modification date. * Log performance of clearing files by extension * Refactor components to provide file extension list Parent functionality such as determining the most recent file modified, getting the last modified date, and checking for alternate source files is better done by having the class provide the list of file extensions that might be used by the class, and having single external functions perform these tasks. (Avoids some redundant code.) * Add multi-file support to file properties hash This will allow us to more accurately detect changes in non-primary source files. (Such as a change in a shared image.) * Simplify component class Removing three functions that are not uniquely specific to each component type and are handled by external functions now that we have exposed the source code file extensions. * Remove Upgrade function on IDbComponent We don't need to try to support mixes of various versions of export files. Use the same version of VCS to build a fresh copy of the project, then export with the latest VCS to upgrade source file. * Move logic to clear legacy files Moving this to the Export function. * Rework processing of conflicts & orphaned objects Refactored the detection and processing of source conflicts and orphaned source files & database objects to better handle source file types that involve multiple files. joyfullservice#473 joyfullservice#471 joyfullservice#472 * Compare source contents of related files When checking for changes in source files, we need to check all the related source files for each component, not just the primary source file. * Adapt export comparison to support multiple files Further changes to compare all related source files. * Update testing database Updated to latest version of VCS. * Include class instancing in code module hash Class modules have an instancing property that needs to be checked for changes along with the VBA code to ensure that the database object matches the last export. A module will now be flagged as changed if the instancing property is changed. * Trap any XML import errors * Add alternate XML format function for big files Large XML files may cause memory errors with XSLT operations. Adding an alternate approach to simply replace the leading tabs with two spaces. This should allow the add-in to export even extremely large table data files as formatted XML. joyfullservice#389, fixes joyfullservice#474 * Remove format version from custom groups Any recent version of export file should be using the new format, and we don't need to carry this conversion forward into v4. * Fix issue with orphaned file detection Need to pass a dictionary, not a collection to the CompareToIndex function. * Move testing code to testing module * Require hash on index update Any update to the index is now required to provide a hash to match the source file. joyfullservice#472 * Save schema filter rules as collection Saving each filter line as a single element in a collection makes a much more readable section in the options.json file, especially when the rules become more complex. Previously this was saved as a combined string value which makes it harder to read changes to individual rules. * Support AfterBuild hooks in add-in project Made a tweak so we can use the RunAfterBuild hook in the add-in project to verify (load) the resources immediately after building from source. This will help prevent accidentally deploying the add-in without the needed resource records, as happened in joyfullservice#477. * Rename as new files * Restore original files * Split forms from VBA code in add-in project Going forward, this will allow us to edit the VBA code without affecting the layout definition files in forms. * Add region support type double (joyfullservice#481) Co-authored-by: Festiis <festim.nuredini@axami.se> * Add some additional comments to code changes Clarifies why we are using the Val() function when parsing ISO dates. * Add initial support for CommandBar popup menus This is still a work in progress, but has the basic functionality of exporting and importing custom CommandBars. * Add error handling to linked table refresh This could be related to a recent Access bug, but it is helpful to trap the error if it occurs. joyfullservice#484 --------- Co-authored-by: Hecon5 <54177882+hecon5@users.noreply.github.com> Co-authored-by: joyfullservice <joyfullservice@users.noreply.github.com> Co-authored-by: bclothier <bgclothier@gmail.com> Co-authored-by: Tanarri <Tanarri@users.noreply.github.com> Co-authored-by: Festim Nuredini <44016065+Festiis@users.noreply.github.com> Co-authored-by: Festiis <festim.nuredini@axami.se>
Windows 10 provides a "Beta" option to attempt UTF-8 support for programs that may not natively support it. You can read more about this option here.
We have seen this issue in the past (#60, #186, #180, #246, #377) but I am wondering if we might be able to resolve this by properly converting the encoding of the exported file to a standard UTF-8 BOM encoding so that it can be universally used by computers with or without the Beta option.
The goal here is to identify the native encoding of the raw export file, and convert it to the correct format before sanitizing and saving the content as UTF-8 in the final source file.
@ollypsilon - Would you be able to attach a sample form, exported with the UTF-8 Beta option active? Please use the following command directly in the immediate window to export the form in the default, unsanitized format, just exactly as it comes from Access.
SaveAsText acForm, "Form1", "Form1.bas"
This will allow me to do some testing on various encoding conversions to see what we need to use for this file format.
It would also be helpful to know what happens if in the same database, you delete the
Form1
object, then run the following command to attempt to import it back into the database. Does it import successfully and correctly? This will give us some clues on the format that Access is expecting for the import when using the UTF-8 Beta option in Windows.LoadFromText acForm, "Form1", "Form1.bas"
The text was updated successfully, but these errors were encountered: