diff options
236 files changed, 10802 insertions, 1184 deletions
diff --git a/Changelog.txt b/Changelog.txt index c4a33386..dc7b5ab5 100755 --- a/Changelog.txt +++ b/Changelog.txt @@ -1,4 +1,27 @@ -FreeFileSync 10.8 [2019-01-19] +FreeFileSync 10.9 [2019-02-10] +------------------------------ +Added FTP, SFTP, Google Drive support for Linux +FreeFileSync Donation Edition available for Linux +Compress file stream during Google Drive upload +Navigate beyond access-denied parents in SFTP folder picker +Fixed unexpected stream size error during FTP upload +Support native recursive deletion for Google Drive +Support native recursive deletion for MTP +Deterministically save Google Drive state during exit +Work around missing TMPDIR variable (Linux) +Support SFTP servers returning large package sizes during folder reading +Start with home path when using SFTP folder picker +Aggregate device authentication prompts during comparison +Clean up temp file after unexpected stream size error +Work around FTP servers not supporting HELP command +Support parsing path by volume name when volume is missing +Parse and streamline Google Drive error messages +Load next item after deleting from config history +Avoid redundant Google Drive syncs after file/folder creation +Avoid duplicate MTP item creation by multiple threads + + +FreeFileSync 10.8 [2019-01-15] ------------------------------ Support synchronization with Google Drive Don't reset sync directions when changing versioning or deletion handling @@ -73,7 +96,7 @@ Fixed statistics boxes background distortion (macOS) FreeFileSync 10.4 [2018-09-09] ------------------------------ -Allow overriding log folder path for gui and batch runs +Allow overriding log folder path for GUI and batch runs Fixed RealTimeSync not triggering when using volume path by name Fixed reading FTP folders including wildcard chars Fixed image overlay graphics glitch (Linux) @@ -261,7 +284,7 @@ Fixed out of memory error when copying large files via FTP New popup dialog option to ignore all errors Reduced memory peaks by enforcing streaming buffer size limits Removed custom sync directions from config XML if not needed -Fixed EOPNOTSUPP error on gvfs-mounted FTP (Linux) +Fixed EOPNOTSUPP error on GVFS-mounted FTP (Linux) Prevent input focus stealing after manual comparison Flash task bar after comparison if other app has input focus @@ -422,7 +445,7 @@ Support nanosecond-precision file time copying (Linux) Start maximized instead of in full screen mode (OS X) Fixed crash while setting privileges during shutdown Fixed crash when failing to clean up log files -Fixed EOPNOTSUPP error when copying file to gvfs Samba share (Linux) +Fixed EOPNOTSUPP error when copying file to GVFS Samba share (Linux) Fixed default external applications command line (Linux) Thread-safe translation access and change during app shutdown Don't consider port and password when comparing SFTP paths @@ -455,7 +478,7 @@ Fixed crash on startup due to missing root certificates Work around start up crash on Windows installations missing certain patches Fixed in-place progress panel height being trimmed Support drawing arbitrary polygons with graph control -Apply Posix file name normalization (OS X) +Apply POSIX file name normalization (OS X) Normalize keyboard input encoding for all text fields (OS X) Report errors when cleaning up old log files Integrate external app WinMerge if installation is found @@ -469,7 +492,7 @@ Log info about non-default global settings Establish new network connections only when needed (Windows) Show only a single login dialog per network share Show login dialogs for the same network address one after another -Fixed endless recursion for paths containing certain unicode characters (OS X) +Fixed endless recursion for paths containing certain Unicode characters (OS X) Support using portable version without direct installation Fixed access denied error when verifying read-only target file (Windows) New global option for sound cue after comparison @@ -559,7 +582,7 @@ Support MTP devices over WiFi with null modification times Do not apply invalid vertical main dialog positions (OS X) Support Yosemite full screen window mode (OS X) Use buffered lock file I/O (Windows) -Correctly setup OpenSSL for multithreaded use +Correctly set up OpenSSL for multithreaded use Added COM initialization for worker threads (Windows) Forward focus to sync button after comparison Streamlined file system abstraction layer interfaces @@ -608,7 +631,7 @@ Added option to set non-standard SFTP port Prevent recursive creation of temporary Recycle Bin directories (Windows) Retrieve grid column label colors from the system Fixed detection of already existing files when moving (Linux) -Follow os convention for preferences (OS X) +Follow OS convention for preferences (OS X) Prevent progress dialog from hiding behind main dialog (OS X) Fixed config saved status not updating when changing certain settings Support for high dpi display settings @@ -789,7 +812,7 @@ Ignore leading/trailing whitespace in search panel Disable search panel during comparison Disable shortkeys during comparison Log folder pair only if files are synced -Fixed number separator formatting for english locale +Fixed number separator formatting for English locale Copying locked files now inactive by default Show all affected folders when warning about a shared sub folder @@ -982,7 +1005,7 @@ FreeFileSync 5.21 [2013-09-02] Detect moved/renamed files in mirror and custom variants New database format for two way variant: old database files are converted automatically Support double-clicking ffs_gui/ffs_batch files (OS X) -Integrated search panel (Ctrl + F, F3) into main dialog +Integrated search panel (CTRL + F, F3) into main dialog Merged variant names into top button labels Hide dock icon while minimized to notification area (OS X) New keyboard shortcuts: F5, F6, F7, F8, F9, F10 @@ -1141,7 +1164,7 @@ Implemented file icon support for sync preview (OS X) RealTimeSync exit via menu working again Restore main dialog even if "close progress dialog" is selected Show full path when failing to create directory on not existing target drive -Middle grid tool tip shown correctly again (Suse Linux/X11) +Middle grid tool tip shown correctly again (SUSE Linux/X11) Prevent process hang when manually writing to directory history (Linux and OS X, wxWidgets 2.9.4) Resolved crash after showing help dialog (OS X) Properly handle non-ASCII characters for external commands (OS X) @@ -1389,7 +1412,7 @@ Fixed "Windows Error Code 59: An unexpected network error occurred" New filter pattern: *\* matches all files in sub directories of base directories Fixed "*?" filter sub-sequence Fixed "Cannot convert from the charset 'Unknown encoding (-1)'!" -Support Ctrl + A in filter dialog +Support CTRL + A in filter dialog Support large filter lists > 32 kByte Allow to hide file icons Avoid switching monitor when main dialog is maximized on multiple monitor systems @@ -1921,7 +1944,7 @@ Support for relative directory names (e.g. \foo, ..\bar) respecting current work New tool tip in middle grid showing detailed information (including conflicts) Status feedback and new abort button for manual deletion Options to add/remove folder pairs in batch dialog -Added tool tip showing progress for silent batchmode +Added tool tip showing progress for silent batch mode New view filter buttons in synchronization preview Revisioned handling of symbolic links (Linux/Windows) GUI optimizations removing flicker @@ -2051,7 +2074,7 @@ Support for \\?\ path prefix for unrestricted path length (directory names > 255 Copy files even if target folder does not exist Fixed occasional error when switching languages Added sys-tray icon for silent batch mode (pause, abort, about) -Support for numeric del-key +Support for numeric DEL-key Avoid endless loops with Vista symbolic links (don't traverse into symbolic links - configurable) New functionality for loading batch files (load button or drag & drop to main/batch window) New options for batch file error handling: "pop up, ignore errors, exit with returncode < 0" @@ -2169,7 +2192,7 @@ FreeFileSync 1.9 [2008-10-26] ----------------------------- Fixed wxWidgets multithreading issue that could cause synchronization to hang occasionally Fixed issue with %1 parameter -Fixed issue with recycle bin usage in unicode mode +Fixed issue with recycle bin usage in Unicode mode Added uninstaller New installer option to associate *.ffs files with FreeFileSync Transformed language files to Unicode (UTF-8) @@ -2281,7 +2304,7 @@ Allow direct keyboard input for directory names Added possibility to continue on error Added indicator for sort direction Simplified code concerning loading of UI resources -Prepared code to support unicode in some future version +Prepared code to support Unicode in some future version Updated German translation diff --git a/FreeFileSync/Build/Languages/german.lng b/FreeFileSync/Build/Languages/german.lng index 0a945231..e89cb5e3 100755 --- a/FreeFileSync/Build/Languages/german.lng +++ b/FreeFileSync/Build/Languages/german.lng @@ -815,6 +815,9 @@ Die Befehlszeile wird ausgelöst, wenn: <source>Loading...</source> <target>Lade...</target> +<source>Scanning...</source> +<target>Suche Dateien...</target> + <source>job name</source> <target>Auftragsname</target> @@ -1255,9 +1258,6 @@ Die Befehlszeile wird ausgelöst, wenn: <source>Detect server limit</source> <target>Ermittle Serverlimit</target> -<source>Select a directory on the server:</source> -<target>Verzeichnis auf dem Server auswählen:</target> - <source>Select Folder</source> <target>Ordner auswählen</target> @@ -1699,9 +1699,6 @@ Dadurch wird ein konsistenter Datenstand auch bei schweren Fehlern garantiert. <source>Initializing...</source> <target>Initialisiere...</target> -<source>Scanning...</source> -<target>Suche Dateien...</target> - <source>Comparing content...</source> <target>Vergleiche Dateiinhalt...</target> diff --git a/FreeFileSync/Build/Resources.zip b/FreeFileSync/Build/Resources.zip Binary files differindex 9aab102a..b3146ea1 100755 --- a/FreeFileSync/Build/Resources.zip +++ b/FreeFileSync/Build/Resources.zip diff --git a/FreeFileSync/Source/Makefile b/FreeFileSync/Source/Makefile index 1e581964..6c03cb56 100755 --- a/FreeFileSync/Source/Makefile +++ b/FreeFileSync/Source/Makefile @@ -1,19 +1,25 @@ -APPNAME = FreeFileSync -prefix = /usr -BINDIR = $(DESTDIR)$(prefix)/bin -SHAREDIR = $(DESTDIR)$(prefix)/share -APPSHAREDIR = $(SHAREDIR)/$(APPNAME) -DOCSHAREDIR = $(SHAREDIR)/doc/$(APPNAME) - -CXXFLAGS = -std=c++17 -pipe -DWXINTL_NO_GETTEXT_MACRO -I../.. -I../../zenXml -include "zen/i18n.h" -include "zen/warn_static.h" \ +EXENAME = FreeFileSync_$(shell arch) + +CXXFLAGS = -std=c++17 -pipe -DWXINTL_NO_GETTEXT_MACRO -DLIBSSH2_OPENSSL -I../.. -I../../zenXml -include "zen/i18n.h" -include "zen/warn_static.h" \ -Wall -Wfatal-errors -Wmissing-include-dirs -Wswitch-enum -Wcast-align -Wshadow -Wnon-virtual-dtor \ -O3 -DNDEBUG `wx-config --cxxflags --debug=no` -pthread LINKFLAGS = -s -no-pie `wx-config --libs std, aui --debug=no` -pthread -#Gtk - support recycler/icon loading/no button border/grid scrolling + +CXXFLAGS += `pkg-config --cflags openssl` +LINKFLAGS += `pkg-config --libs openssl` + +CXXFLAGS += `pkg-config --cflags libcurl` +LINKFLAGS += `pkg-config --libs libcurl` + +CXXFLAGS += `pkg-config --cflags libssh2` +LINKFLAGS += `pkg-config --libs libssh2` + CXXFLAGS += `pkg-config --cflags gtk+-2.0` LINKFLAGS += `pkg-config --libs gtk+-2.0` +#treat as system headers so that warnings are hidden: +CXXFLAGS += -isystem/usr/include/gtk-2.0 #support for SELinux (optional) SELINUX_EXISTING=$(shell pkg-config --exists libselinux && echo YES) @@ -53,8 +59,15 @@ CPP_FILES+=base/synchronization.cpp CPP_FILES+=base/versioning.cpp CPP_FILES+=fs/abstract.cpp CPP_FILES+=fs/concrete.cpp +CPP_FILES+=fs/ftp.cpp +CPP_FILES+=fs/gdrive.cpp +CPP_FILES+=fs/init_curl_libssh2.cpp CPP_FILES+=fs/native.cpp +CPP_FILES+=fs/sftp.cpp +CPP_FILES+=fs/libssh2/init_libssh2.cpp +CPP_FILES+=fs/libssh2/init_open_ssl.cpp CPP_FILES+=ui/batch_config.cpp +CPP_FILES+=ui/abstract_folder_picker.cpp CPP_FILES+=ui/batch_status_handler.cpp CPP_FILES+=ui/cfg_grid.cpp CPP_FILES+=ui/command_box.cpp @@ -96,34 +109,20 @@ CPP_FILES+=../../wx+/popup_dlg.cpp CPP_FILES+=../../wx+/popup_dlg_generated.cpp CPP_FILES+=../../xBRZ/src/xbrz.cpp -OBJ_FILES = $(CPP_FILES:%=../Obj/FFS_GCC_Make_Release/ffs/src/%.o) +TMP_PATH = /tmp/$(EXENAME)_Make + +OBJ_FILES = $(CPP_FILES:%=$(TMP_PATH)/ffs/src/%.o) -all: ../Build/Bin/$(APPNAME) +all: ../Build/Bin/$(EXENAME) -../Build/Bin/$(APPNAME): $(OBJ_FILES) +../Build/Bin/$(EXENAME): $(OBJ_FILES) + mkdir -p $(dir $@) g++ -o $@ $^ $(LINKFLAGS) -../Obj/FFS_GCC_Make_Release/ffs/src/%.o : % +$(TMP_PATH)/ffs/src/%.o : % mkdir -p $(dir $@) g++ $(CXXFLAGS) -c $< -o $@ clean: - rm -rf ../Obj/FFS_GCC_Make_Release - rm -f ../Build/Bin/$(APPNAME) - -install: - mkdir -p $(BINDIR) - cp ../Build/Bin/$(APPNAME) $(BINDIR) - - mkdir -p $(APPSHAREDIR) - cp -R ../Build/Languages/ \ - ../Build/ding.wav \ - ../Build/gong.wav \ - ../Build/harp.wav \ - ../Build/Resources.zip \ - $(APPSHAREDIR) - - mkdir -p $(DOCSHAREDIR) - cp ../../Changelog.txt $(DOCSHAREDIR)/CHANGELOG -# cp "../Build/User Manual.pdf" $(DOCSHAREDIR) - gzip $(DOCSHAREDIR)/CHANGELOG + rm -rf $(TMP_PATH) + rm -f ../Build/Bin/$(EXENAME) diff --git a/FreeFileSync/Source/RealTimeSync/Makefile b/FreeFileSync/Source/RealTimeSync/Makefile index dc129c1c..b4dc8b59 100755 --- a/FreeFileSync/Source/RealTimeSync/Makefile +++ b/FreeFileSync/Source/RealTimeSync/Makefile @@ -1,6 +1,4 @@ -APPNAME = RealTimeSync -prefix = /usr -BINDIR = $(DESTDIR)$(prefix)/bin +EXENAME = RealTimeSync_$(shell arch) CXXFLAGS = -std=c++17 -pipe -DWXINTL_NO_GETTEXT_MACRO -I../../.. -I../../../zenXml -include "zen/i18n.h" -include "zen/warn_static.h" \ -Wall -Wfatal-errors -Wmissing-include-dirs -Wswitch-enum -Wcast-align -Wshadow -Wnon-virtual-dtor \ @@ -11,6 +9,8 @@ LINKFLAGS = -s -no-pie `wx-config --libs std, aui --debug=no` -pthread #Gtk - support "no button border" CXXFLAGS += `pkg-config --cflags gtk+-2.0` LINKFLAGS += `pkg-config --libs gtk+-2.0` +#treat as system headers so that warnings are hidden: +CXXFLAGS += -isystem/usr/include/gtk-2.0 CPP_FILES= CPP_FILES+=application.cpp @@ -20,6 +20,9 @@ CPP_FILES+=tray_menu.cpp CPP_FILES+=monitor.cpp CPP_FILES+=xml_proc.cpp CPP_FILES+=folder_selector2.cpp +CPP_FILES+=../fs/abstract.cpp +CPP_FILES+=../base/icon_buffer.cpp +CPP_FILES+=../base/icon_loader.cpp CPP_FILES+=../base/localization.cpp CPP_FILES+=../base/resolve_path.cpp CPP_FILES+=../base/ffs_paths.cpp @@ -38,21 +41,20 @@ CPP_FILES+=../../../wx+/popup_dlg.cpp CPP_FILES+=../../../wx+/popup_dlg_generated.cpp CPP_FILES+=../../../xBRZ/src/xbrz.cpp -OBJ_FILES=$(CPP_FILES:%=../../Obj/RTS_GCC_Make_Release/ffs/src/rts/%.o) +TMP_PATH = /tmp/$(EXENAME)_Make -all: ../../Build/Bin/$(APPNAME) +OBJ_FILES = $(CPP_FILES:%=$(TMP_PATH)/ffs/src/rts/%.o) -../../Build/Bin/$(APPNAME): $(OBJ_FILES) +all: ../../Build/Bin/$(EXENAME) + +../../Build/Bin/$(EXENAME): $(OBJ_FILES) + mkdir -p $(dir $@) g++ -o $@ $^ $(LINKFLAGS) -../../Obj/RTS_GCC_Make_Release/ffs/src/rts/%.o : % +$(TMP_PATH)/ffs/src/rts/%.o : % mkdir -p $(dir $@) g++ $(CXXFLAGS) -c $< -o $@ clean: - rm -rf ../../Obj/RTS_GCC_Make_Release - rm -f ../../Build/Bin/$(APPNAME) - -install: - mkdir -p $(BINDIR) - cp ../../Build/Bin/$(APPNAME) $(BINDIR) + rm -rf $(TMP_PATH) + rm -f ../../Build/Bin/$(EXENAME) diff --git a/FreeFileSync/Source/RealTimeSync/app_icon.h b/FreeFileSync/Source/RealTimeSync/app_icon.h index 00797e37..00797e37 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/app_icon.h +++ b/FreeFileSync/Source/RealTimeSync/app_icon.h diff --git a/FreeFileSync/Source/RealTimeSync/application.cpp b/FreeFileSync/Source/RealTimeSync/application.cpp index 899d1c69..899d1c69 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/application.cpp +++ b/FreeFileSync/Source/RealTimeSync/application.cpp diff --git a/FreeFileSync/Source/RealTimeSync/application.h b/FreeFileSync/Source/RealTimeSync/application.h index bdfc2027..bdfc2027 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/application.h +++ b/FreeFileSync/Source/RealTimeSync/application.h diff --git a/FreeFileSync/Source/RealTimeSync/folder_selector2.cpp b/FreeFileSync/Source/RealTimeSync/folder_selector2.cpp index 200cdbcf..f32a6f6d 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/folder_selector2.cpp +++ b/FreeFileSync/Source/RealTimeSync/folder_selector2.cpp @@ -43,10 +43,12 @@ void setFolderPath(const Zstring& dirpath, wxTextCtrl* txtCtrl, wxWindow& toolti //############################################################################################################## -FolderSelector2::FolderSelector2(wxWindow& dropWindow, +FolderSelector2::FolderSelector2(wxWindow* parent, + wxWindow& dropWindow, wxButton& selectButton, wxTextCtrl& folderPathCtrl, wxStaticText* staticText) : + parent_(parent), dropWindow_(dropWindow), selectButton_(selectButton), folderPathCtrl_(folderPathCtrl), @@ -140,7 +142,7 @@ void FolderSelector2::onSelectDir(wxCommandEvent& event) } } - wxDirDialog dirPicker(&selectButton_, _("Select a folder"), utfTo<wxString>(defaultFolderPath)); //put modal wxWidgets dialogs on stack: creating on freestore leads to memleak! + wxDirDialog dirPicker(parent_, _("Select a folder"), utfTo<wxString>(defaultFolderPath)); //put modal wxWidgets dialogs on stack: creating on freestore leads to memleak! if (dirPicker.ShowModal() != wxID_OK) return; const Zstring newFolderPath = utfTo<Zstring>(dirPicker.GetPath()); diff --git a/FreeFileSync/Source/RealTimeSync/folder_selector2.h b/FreeFileSync/Source/RealTimeSync/folder_selector2.h index e6af3940..04b2036e 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/folder_selector2.h +++ b/FreeFileSync/Source/RealTimeSync/folder_selector2.h @@ -20,7 +20,8 @@ namespace rts class FolderSelector2 : public wxEvtHandler { public: - FolderSelector2(wxWindow& dropWindow, + FolderSelector2(wxWindow* parent, + wxWindow& dropWindow, wxButton& selectButton, wxTextCtrl& folderPathCtrl, wxStaticText* staticText); //optional @@ -36,6 +37,7 @@ private: void onEditFolderPath(wxCommandEvent& event); void onSelectDir (wxCommandEvent& event); + wxWindow* parent_; wxWindow& dropWindow_; wxButton& selectButton_; wxTextCtrl& folderPathCtrl_; diff --git a/FreeFileSync/Source/RealTimeSync/gui_generated.cpp b/FreeFileSync/Source/RealTimeSync/gui_generated.cpp index aec55f7c..bcd592ac 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/gui_generated.cpp +++ b/FreeFileSync/Source/RealTimeSync/gui_generated.cpp @@ -108,9 +108,18 @@ MainDlgGenerated::MainDlgGenerated( wxWindow* parent, wxWindowID id, const wxStr wxBoxSizer* bSizer151; bSizer151 = new wxBoxSizer( wxVERTICAL ); + wxBoxSizer* bSizer142; + bSizer142 = new wxBoxSizer( wxHORIZONTAL ); + + m_bitmapFolders = new wxStaticBitmap( m_panelMain, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxDefaultSize, 0 ); + bSizer142->Add( m_bitmapFolders, 0, wxTOP|wxBOTTOM|wxLEFT, 5 ); + m_staticText7 = new wxStaticText( m_panelMain, wxID_ANY, _("Folders to watch:"), wxDefaultPosition, wxDefaultSize, 0 ); m_staticText7->Wrap( -1 ); - bSizer151->Add( m_staticText7, 0, wxALL, 5 ); + bSizer142->Add( m_staticText7, 0, wxALL|wxALIGN_CENTER_VERTICAL, 5 ); + + + bSizer151->Add( bSizer142, 0, 0, 5 ); m_panelMainFolder = new wxPanel( m_panelMain, wxID_ANY, wxDefaultPosition, wxDefaultSize, wxTAB_TRAVERSAL ); m_panelMainFolder->SetBackgroundColour( wxSystemSettings::GetColour( wxSYS_COLOUR_WINDOW ) ); @@ -203,14 +212,23 @@ MainDlgGenerated::MainDlgGenerated( wxWindow* parent, wxWindowID id, const wxStr wxBoxSizer* bSizer141; bSizer141 = new wxBoxSizer( wxVERTICAL ); + wxBoxSizer* bSizer13; + bSizer13 = new wxBoxSizer( wxHORIZONTAL ); + + m_bitmapCommand = new wxStaticBitmap( m_panelMain, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxDefaultSize, 0 ); + bSizer13->Add( m_bitmapCommand, 0, wxTOP|wxBOTTOM|wxLEFT|wxALIGN_CENTER_VERTICAL, 5 ); + m_staticText6 = new wxStaticText( m_panelMain, wxID_ANY, _("Command line:"), wxDefaultPosition, wxDefaultSize, 0 ); m_staticText6->Wrap( -1 ); - bSizer141->Add( m_staticText6, 0, wxALL, 5 ); + bSizer13->Add( m_staticText6, 0, wxALL|wxALIGN_CENTER_VERTICAL, 5 ); + + + bSizer141->Add( bSizer13, 0, 0, 5 ); m_textCtrlCommand = new wxTextCtrl( m_panelMain, wxID_ANY, wxEmptyString, wxDefaultPosition, wxDefaultSize, 0 ); m_textCtrlCommand->SetToolTip( _("The command is triggered if:\n- files or subfolders change\n- new folders arrive (e.g. USB stick insert)") ); - bSizer141->Add( m_textCtrlCommand, 0, wxBOTTOM|wxRIGHT|wxLEFT|wxEXPAND, 5 ); + bSizer141->Add( m_textCtrlCommand, 0, wxEXPAND|wxBOTTOM|wxRIGHT|wxLEFT, 5 ); bSizer1->Add( bSizer141, 0, wxALL|wxEXPAND, 5 ); diff --git a/FreeFileSync/Source/RealTimeSync/gui_generated.h b/FreeFileSync/Source/RealTimeSync/gui_generated.h index 5bfb621b..080fd473 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/gui_generated.h +++ b/FreeFileSync/Source/RealTimeSync/gui_generated.h @@ -25,6 +25,7 @@ namespace zen { class BitmapTextButton; } #include <wx/stattext.h> #include <wx/sizer.h> #include <wx/statline.h> +#include <wx/statbmp.h> #include <wx/bmpbuttn.h> #include <wx/button.h> #include <wx/textctrl.h> @@ -57,6 +58,7 @@ protected: wxStaticText* m_staticText811; wxStaticLine* m_staticline2; wxPanel* m_panelMain; + wxStaticBitmap* m_bitmapFolders; wxStaticText* m_staticText7; wxPanel* m_panelMainFolder; wxStaticText* m_staticTextFinalPath; @@ -70,6 +72,7 @@ protected: wxStaticText* m_staticText8; wxSpinCtrl* m_spinCtrlDelay; wxStaticLine* m_staticline211; + wxStaticBitmap* m_bitmapCommand; wxStaticText* m_staticText6; wxTextCtrl* m_textCtrlCommand; wxStaticLine* m_staticline5; diff --git a/FreeFileSync/Source/RealTimeSync/main_dlg.cpp b/FreeFileSync/Source/RealTimeSync/main_dlg.cpp index 7e2a753a..37c04636 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/main_dlg.cpp +++ b/FreeFileSync/Source/RealTimeSync/main_dlg.cpp @@ -18,6 +18,7 @@ #include "tray_menu.h" #include "app_icon.h" #include "../base/help_provider.h" +#include "../base/icon_buffer.h" #include "../base/ffs_paths.h" #include "../version/version.h" @@ -48,7 +49,7 @@ class rts::DirectoryPanel : public FolderGenerated public: DirectoryPanel(wxWindow* parent) : FolderGenerated(parent), - folderSelector_(*this, *m_buttonSelectFolder, *m_txtCtrlDirectory, nullptr /*staticText*/) + folderSelector_(parent, *this, *m_buttonSelectFolder, *m_txtCtrlDirectory, nullptr /*staticText*/) { m_bpButtonRemoveFolder->SetBitmapLabel(getResourceImage(L"item_remove")); } @@ -63,13 +64,13 @@ private: void MainDialog::create(const Zstring& cfgFile) { - /*MainDialog* frame = */ new MainDialog(nullptr, cfgFile); + /*MainDialog* frame = */ new MainDialog(cfgFile); } -MainDialog::MainDialog(wxDialog* dlg, const Zstring& cfgFileName) - : MainDlgGenerated(dlg), - lastRunConfigPath_(fff::getConfigDirPathPf() + Zstr("LastRun.ffs_real")) +MainDialog::MainDialog(const Zstring& cfgFileName) : + MainDlgGenerated(nullptr), + lastRunConfigPath_(fff::getConfigDirPathPf() + Zstr("LastRun.ffs_real")) { SetIcon(getRtsIcon()); //set application icon @@ -83,6 +84,8 @@ MainDialog::MainDialog(wxDialog* dlg, const Zstring& cfgFileName) m_bpButtonRemoveTopFolder->Hide(); m_panelMainFolder->Layout(); + m_bitmapFolders->SetBitmap(fff::IconBuffer::genericDirIcon(fff::IconBuffer::SIZE_SMALL)); + m_bitmapCommand->SetBitmap(shrinkImage(getResourceImage(L"command_line").ConvertToImage(), fastFromDIP(20))); m_bpButtonAddFolder ->SetBitmapLabel(getResourceImage(L"item_add")); m_bpButtonRemoveTopFolder->SetBitmapLabel(getResourceImage(L"item_remove")); setBitmapTextLabel(*m_buttonStart, getResourceImage(L"startRts").ConvertToImage(), m_buttonStart->GetLabel(), fastFromDIP(5), fastFromDIP(8)); @@ -91,7 +94,7 @@ MainDialog::MainDialog(wxDialog* dlg, const Zstring& cfgFileName) Connect(wxEVT_CHAR_HOOK, wxKeyEventHandler(MainDialog::OnKeyPressed), nullptr, this); //prepare drag & drop - firstFolderPanel_ = std::make_unique<FolderSelector2>(*m_panelMainFolder, *m_buttonSelectFolderMain, *m_txtCtrlDirectoryMain, m_staticTextFinalPath); + firstFolderPanel_ = std::make_unique<FolderSelector2>(this, *m_panelMainFolder, *m_buttonSelectFolderMain, *m_txtCtrlDirectoryMain, m_staticTextFinalPath); //--------------------------- load config values ------------------------------------ XmlRealConfig newConfig; diff --git a/FreeFileSync/Source/RealTimeSync/main_dlg.h b/FreeFileSync/Source/RealTimeSync/main_dlg.h index bc2eaacb..d20b889c 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/main_dlg.h +++ b/FreeFileSync/Source/RealTimeSync/main_dlg.h @@ -31,7 +31,7 @@ public: void onQueryEndSession(); //last chance to do something useful before killing the application! private: - MainDialog(wxDialog* dlg, const Zstring& cfgFileName); + MainDialog(const Zstring& cfgFileName); ~MainDialog(); void loadConfig(const Zstring& filepath); diff --git a/FreeFileSync/Source/RealTimeSync/monitor.cpp b/FreeFileSync/Source/RealTimeSync/monitor.cpp index 647a1114..5d103799 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/monitor.cpp +++ b/FreeFileSync/Source/RealTimeSync/monitor.cpp @@ -26,7 +26,7 @@ std::set<Zstring, LessNativePath> waitForMissingDirs(const std::vector<Zstring>& const std::function<void(const Zstring& folderPath)>& requestUiRefresh, std::chrono::milliseconds cbInterval) { //early failure! check for unsupported folder paths: - for (const Zstring& protoName : { Zstr("FTP"), Zstr("SFTP"), Zstr("MTP") }) + for (const Zstring& protoName : { Zstr("ftp"), Zstr("sftp"), Zstr("mtp"), Zstr("gdrive") }) for (const Zstring& phrase : folderPathPhrases) //hopefully clear enough now: https://freefilesync.org/forum/viewtopic.php?t=4302 if (startsWithAsciiNoCase(trimCpy(phrase), protoName + Zstr(":"))) diff --git a/FreeFileSync/Source/RealTimeSync/monitor.h b/FreeFileSync/Source/RealTimeSync/monitor.h index 06d01161..06d01161 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/monitor.h +++ b/FreeFileSync/Source/RealTimeSync/monitor.h diff --git a/FreeFileSync/Source/RealTimeSync/tray_menu.cpp b/FreeFileSync/Source/RealTimeSync/tray_menu.cpp index 106c508b..106c508b 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/tray_menu.cpp +++ b/FreeFileSync/Source/RealTimeSync/tray_menu.cpp diff --git a/FreeFileSync/Source/RealTimeSync/tray_menu.h b/FreeFileSync/Source/RealTimeSync/tray_menu.h index 79c63dc2..79c63dc2 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/tray_menu.h +++ b/FreeFileSync/Source/RealTimeSync/tray_menu.h diff --git a/FreeFileSync/Source/RealTimeSync/xml_proc.cpp b/FreeFileSync/Source/RealTimeSync/xml_proc.cpp index 4cd8636f..4cd8636f 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/xml_proc.cpp +++ b/FreeFileSync/Source/RealTimeSync/xml_proc.cpp diff --git a/FreeFileSync/Source/RealTimeSync/xml_proc.h b/FreeFileSync/Source/RealTimeSync/xml_proc.h index ebad3c7e..ebad3c7e 100755..100644 --- a/FreeFileSync/Source/RealTimeSync/xml_proc.h +++ b/FreeFileSync/Source/RealTimeSync/xml_proc.h diff --git a/FreeFileSync/Source/base/algorithm.cpp b/FreeFileSync/Source/base/algorithm.cpp index b04cbb36..b04cbb36 100755..100644 --- a/FreeFileSync/Source/base/algorithm.cpp +++ b/FreeFileSync/Source/base/algorithm.cpp diff --git a/FreeFileSync/Source/base/algorithm.h b/FreeFileSync/Source/base/algorithm.h index a58e2b50..a58e2b50 100755..100644 --- a/FreeFileSync/Source/base/algorithm.h +++ b/FreeFileSync/Source/base/algorithm.h diff --git a/FreeFileSync/Source/base/application.cpp b/FreeFileSync/Source/base/application.cpp index 8f156b85..678454ac 100755..100644 --- a/FreeFileSync/Source/base/application.cpp +++ b/FreeFileSync/Source/base/application.cpp @@ -27,6 +27,7 @@ #include "../ui/main_dlg.h" #include <gtk/gtk.h> + #include "../fs/concrete.h" using namespace zen; @@ -76,6 +77,7 @@ bool Application::OnInit() } catch (FileError&) { assert(false); } + initAfs({ getResourceDirPf(), getConfigDirPathPf() }); //also inits OpenSSL (used in runSanityChecks() on Linux) Connect(wxEVT_QUERY_END_SESSION, wxEventHandler(Application::onQueryEndSession), nullptr, this); @@ -93,6 +95,7 @@ int Application::OnExit() uninitializeHelp(); releaseWxLocale(); cleanupResourceImages(); + teardownAfs(); //throw FileError return wxApp::OnExit(); } @@ -169,16 +172,11 @@ void Application::launch(const std::vector<Zstring>& commandArgs) Zstring globalConfigFile; bool openForEdit = false; { - std::vector<Zstring> dirPathPhrasesLeft; //TODO: remove migration code at some time! 2017-12-14 - std::vector<Zstring> dirPathPhrasesRight; // + const Zchar* optionEdit = Zstr("-edit"); + const Zchar* optionDirPair = Zstr("-dirpair"); + const Zchar* optionSendTo = Zstr("-sendto"); //remaining arguments are unspecified number of folder paths; wonky syntax; let's keep it undocumented - const Zchar optionEdit [] = Zstr("-edit"); - const Zchar optionLeftDir [] = Zstr("-leftdir"); //TODO: remove migration code at some time! 2017-12-14 - const Zchar optionRightDir[] = Zstr("-rightdir"); // - const Zchar optionDirPair [] = Zstr("-dirpair"); - const Zchar optionSendTo [] = Zstr("-sendto"); //remaining arguments are unspecified number of folder paths; wonky syntax; let's keep it undocumented - - auto syntaxHelpRequested = [&](const Zstring& arg) + auto isHelpRequest = [](const Zstring& arg) { auto it = std::find_if(arg.begin(), arg.end(), [](Zchar c) { return c != Zstr('/') && c != Zstr('-'); }); if (it == arg.begin()) return false; //require at least one prefix character @@ -191,31 +189,17 @@ void Application::launch(const std::vector<Zstring>& commandArgs) auto isCommandLineOption = [&](const Zstring& arg) { - return equalAsciiNoCase(arg, optionEdit ) || - equalAsciiNoCase(arg, optionLeftDir ) || - equalAsciiNoCase(arg, optionRightDir) || - equalAsciiNoCase(arg, optionDirPair ) || - equalAsciiNoCase(arg, optionSendTo ) || - syntaxHelpRequested(arg); + return equalAsciiNoCase(arg, optionEdit ) || + equalAsciiNoCase(arg, optionDirPair) || + equalAsciiNoCase(arg, optionSendTo ) || + isHelpRequest(arg); }; for (auto it = commandArgs.begin(); it != commandArgs.end(); ++it) - if (syntaxHelpRequested(*it)) + if (isHelpRequest(*it)) return showSyntaxHelp(); else if (equalAsciiNoCase(*it, optionEdit)) openForEdit = true; - else if (equalAsciiNoCase(*it, optionLeftDir)) - { - if (++it == commandArgs.end() || isCommandLineOption(*it)) - return notifyFatalError(replaceCpy(_("A directory path is expected after %x."), L"%x", utfTo<std::wstring>(optionLeftDir)), _("Syntax error")); - dirPathPhrasesLeft.push_back(*it); - } - else if (equalAsciiNoCase(*it, optionRightDir)) - { - if (++it == commandArgs.end() || isCommandLineOption(*it)) - return notifyFatalError(replaceCpy(_("A directory path is expected after %x."), L"%x", utfTo<std::wstring>(optionRightDir)), _("Syntax error")); - dirPathPhrasesRight.push_back(*it); - } else if (equalAsciiNoCase(*it, optionDirPair)) { if (++it == commandArgs.end() || isCommandLineOption(*it)) @@ -301,12 +285,6 @@ void Application::launch(const std::vector<Zstring>& commandArgs) return notifyFatalError(e.toString(), _("Error")); } } - - if (dirPathPhrasesLeft.size() != dirPathPhrasesRight.size()) - return notifyFatalError(_("Unequal number of left and right directories specified."), _("Syntax error")); - - for (size_t i = 0; i < dirPathPhrasesLeft.size(); ++i) - dirPathPhrasePairs.emplace_back(dirPathPhrasesLeft[i], dirPathPhrasesRight[i]); } //---------------------------------------------------------------------------------------------------- @@ -529,7 +507,6 @@ void runBatchMode(const Zstring& globalConfigFilePath, const XmlBatchConfig& bat // checkForUpdatePeriodically(globalCfg.lastUpdateCheck); //WinInet not working when FFS is running as a service!!! https://support.microsoft.com/en-us/kb/238425 - const std::map<AfsDevice, size_t>& deviceParallelOps = batchCfg.mainCfg.deviceParallelOps; std::set<AbstractPath> logFilePathsToKeep; for (const ConfigFileItem& item : globalCfg.gui.mainDlg.cfgFileHistory) @@ -566,7 +543,6 @@ void runBatchMode(const Zstring& globalConfigFilePath, const XmlBatchConfig& bat globalCfg.createLockFile, dirLocks, extractCompareCfg(batchCfg.mainCfg), - deviceParallelOps, statusHandler); //throw AbortProcess //START SYNCHRONIZATION synchronize(syncStartTime, @@ -577,7 +553,6 @@ void runBatchMode(const Zstring& globalConfigFilePath, const XmlBatchConfig& bat globalCfg.runWithBackgroundPriority, extractSyncCfg(batchCfg.mainCfg), cmpResult, - deviceParallelOps, globalCfg.warnDlgs, statusHandler); //throw AbortProcess } diff --git a/FreeFileSync/Source/base/application.h b/FreeFileSync/Source/base/application.h index c08491c8..c08491c8 100755..100644 --- a/FreeFileSync/Source/base/application.h +++ b/FreeFileSync/Source/base/application.h diff --git a/FreeFileSync/Source/base/binary.cpp b/FreeFileSync/Source/base/binary.cpp index db5dee54..db5dee54 100755..100644 --- a/FreeFileSync/Source/base/binary.cpp +++ b/FreeFileSync/Source/base/binary.cpp diff --git a/FreeFileSync/Source/base/binary.h b/FreeFileSync/Source/base/binary.h index 66d8b09c..66d8b09c 100755..100644 --- a/FreeFileSync/Source/base/binary.h +++ b/FreeFileSync/Source/base/binary.h diff --git a/FreeFileSync/Source/base/cmp_filetime.h b/FreeFileSync/Source/base/cmp_filetime.h index 385fba6f..385fba6f 100755..100644 --- a/FreeFileSync/Source/base/cmp_filetime.h +++ b/FreeFileSync/Source/base/cmp_filetime.h diff --git a/FreeFileSync/Source/base/comparison.cpp b/FreeFileSync/Source/base/comparison.cpp index cd873c87..f560725e 100755..100644 --- a/FreeFileSync/Source/base/comparison.cpp +++ b/FreeFileSync/Source/base/comparison.cpp @@ -68,7 +68,7 @@ struct ResolvedBaseFolders }; -ResolvedBaseFolders initializeBaseFolders(const std::vector<FolderPairCfg>& fpCfgList, const std::map<AfsDevice, size_t>& deviceParallelOps, +ResolvedBaseFolders initializeBaseFolders(const std::vector<FolderPairCfg>& fpCfgList, bool allowUserInteraction, WarningDialogs& warnings, ProcessCallback& callback /*throw X*/) @@ -93,7 +93,7 @@ ResolvedBaseFolders initializeBaseFolders(const std::vector<FolderPairCfg>& fpCf allFolders.insert(output.resolvedPairs.back().folderPathRight); } //--------------------------------------------------------------------------- - const FolderStatus status = getFolderStatusNonBlocking(allFolders, deviceParallelOps, //re-check *all* directories on each try! + const FolderStatus status = getFolderStatusNonBlocking(allFolders, allowUserInteraction, callback); //throw X output.existingBaseFolders = status.existing; @@ -147,8 +147,6 @@ ResolvedBaseFolders initializeBaseFolders(const std::vector<FolderPairCfg>& fpCf callback.reportWarning(msg, warnings.warnFoldersDifferInCase); //throw X } - warn_static("maybe just use GetFinalPathNameByHandleW/realpath? (limitation: network folders that deny access to parents)") - return output; } @@ -158,7 +156,6 @@ class ComparisonBuffer { public: ComparisonBuffer(const std::set<DirectoryKey>& foldersToRead, - const std::map<AfsDevice, size_t>& deviceParallelOps, int fileTimeTolerance, ProcessCallback& callback); @@ -179,15 +176,14 @@ private: std::map<DirectoryKey, DirectoryValue> directoryBuffer_; //contains only *existing* directories const int fileTimeTolerance_; ProcessCallback& cb_; - const std::map<AfsDevice, size_t> deviceParallelOps_; }; ComparisonBuffer::ComparisonBuffer(const std::set<DirectoryKey>& foldersToRead, - const std::map<AfsDevice, size_t>& deviceParallelOps, int fileTimeTolerance, ProcessCallback& callback) : - fileTimeTolerance_(fileTimeTolerance), cb_(callback), deviceParallelOps_(deviceParallelOps) + fileTimeTolerance_(fileTimeTolerance), + cb_(callback) { auto onError = [&](const std::wstring& msg, size_t retryNumber) { @@ -216,7 +212,6 @@ ComparisonBuffer::ComparisonBuffer(const std::set<DirectoryKey>& foldersToRead, parallelDeviceTraversal(foldersToRead, //in directoryBuffer_, //out - deviceParallelOps, onError, onStatusUpdate, //throw X UI_UPDATE_INTERVAL / 2); //every ~50 ms @@ -520,8 +515,6 @@ std::list<std::shared_ptr<BaseFolderPair>> ComparisonBuffer::compareByContent(co struct ParallelOps { size_t current = 0; - size_t effectiveMax = 0; //a folder pair is allowed to use the maximum parallelOps that left/right devices support - // => consider max over all folder pairs, that a device is involved with! }; std::map<AfsDevice, ParallelOps> parallelOpsStatus; @@ -535,16 +528,8 @@ std::list<std::shared_ptr<BaseFolderPair>> ComparisonBuffer::compareByContent(co auto addToBinaryWorkload = [&](const AbstractPath& basePathL, const AbstractPath& basePathR, RingBuffer<FilePair*>&& filesToCompareBytewise) { - //calculate effective max parallelOps that devices must support - const size_t parallelOpsFp = std::max(getDeviceParallelOps(deviceParallelOps_, basePathL.afsDevice), - getDeviceParallelOps(deviceParallelOps_, basePathR.afsDevice)); - ParallelOps& posL = parallelOpsStatus[basePathL.afsDevice]; ParallelOps& posR = parallelOpsStatus[basePathR.afsDevice]; - - posL.effectiveMax = std::max(posL.effectiveMax, parallelOpsFp); - posR.effectiveMax = std::max(posR.effectiveMax, parallelOpsFp); - fpWorkload.push_back({ posL, posR, std::move(filesToCompareBytewise) }); }; @@ -616,11 +601,9 @@ std::list<std::shared_ptr<BaseFolderPair>> ComparisonBuffer::compareByContent(co for (size_t j = 0; j < fpWorkload.size(); ++j) { BinaryWorkload& bwl = fpWorkload[j]; - ParallelOps& posL = bwl.parallelOpsL; ParallelOps& posR = bwl.parallelOpsR; - - const size_t newTaskCount = std::min<size_t>({ posL.effectiveMax - posL.current, posR.effectiveMax - posR.current, bwl.filesToCompareBytewise.size() }); + const size_t newTaskCount = std::min<size_t>({ 1 - posL.current, 1 - posR.current, bwl.filesToCompareBytewise.size() }); if (&posL != &posR) posL.current += newTaskCount; // /**/ posR.current += newTaskCount; //consider aliasing! @@ -642,10 +625,6 @@ std::list<std::shared_ptr<BaseFolderPair>> ComparisonBuffer::compareByContent(co bwl.filesToCompareBytewise.pop_front(); } - - assert(0 <= posL.current && posL.current <= posL.effectiveMax); - assert(0 <= posR.current && posR.current <= posR.effectiveMax); - if (posL.current != 0 || posR.current != 0 || !bwl.filesToCompareBytewise.empty()) wereDone = false; } @@ -1039,7 +1018,6 @@ FolderComparison fff::compare(WarningDialogs& warnings, bool createDirLocks, std::unique_ptr<LockHolder>& dirLocks, const std::vector<FolderPairCfg>& fpCfgList, - const std::map<AfsDevice, size_t>& deviceParallelOps, ProcessCallback& callback) { //PERF_START; @@ -1074,7 +1052,8 @@ FolderComparison fff::compare(WarningDialogs& warnings, callback.reportInfo(e.toString()); //throw X } - const ResolvedBaseFolders& resInfo = initializeBaseFolders(fpCfgList, deviceParallelOps, allowUserInteraction, warnings, callback); //throw X + const ResolvedBaseFolders& resInfo = initializeBaseFolders(fpCfgList, + allowUserInteraction, warnings, callback); //throw X //directory existence only checked *once* to avoid race conditions! if (resInfo.resolvedPairs.size() != fpCfgList.size()) throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); @@ -1157,7 +1136,8 @@ FolderComparison fff::compare(WarningDialogs& warnings, { //------------ traverse/read folders ----------------------------------------------------- //PERF_START; - ComparisonBuffer cmpBuff(foldersToRead, deviceParallelOps, fileTimeTolerance, callback); + ComparisonBuffer cmpBuff(foldersToRead, + fileTimeTolerance, callback); //PERF_STOP; //process binary comparison as one junk diff --git a/FreeFileSync/Source/base/comparison.h b/FreeFileSync/Source/base/comparison.h index 82e73637..55935eae 100755..100644 --- a/FreeFileSync/Source/base/comparison.h +++ b/FreeFileSync/Source/base/comparison.h @@ -58,7 +58,6 @@ FolderComparison compare(WarningDialogs& warnings, bool createDirLocks, std::unique_ptr<LockHolder>& dirLocks, //out const std::vector<FolderPairCfg>& fpCfgList, - const std::map<AfsDevice, size_t>& deviceParallelOps, ProcessCallback& callback); } diff --git a/FreeFileSync/Source/base/db_file.cpp b/FreeFileSync/Source/base/db_file.cpp index 9b83d194..9b83d194 100755..100644 --- a/FreeFileSync/Source/base/db_file.cpp +++ b/FreeFileSync/Source/base/db_file.cpp diff --git a/FreeFileSync/Source/base/db_file.h b/FreeFileSync/Source/base/db_file.h index e3e4b4d3..e3e4b4d3 100755..100644 --- a/FreeFileSync/Source/base/db_file.h +++ b/FreeFileSync/Source/base/db_file.h diff --git a/FreeFileSync/Source/base/dir_exist_async.h b/FreeFileSync/Source/base/dir_exist_async.h index 097732fc..db4b1814 100755..100644 --- a/FreeFileSync/Source/base/dir_exist_async.h +++ b/FreeFileSync/Source/base/dir_exist_async.h @@ -32,7 +32,7 @@ struct FolderStatus }; -FolderStatus getFolderStatusNonBlocking(const std::set<AbstractPath>& folderPaths, const std::map<AfsDevice, size_t>& deviceParallelOps, +FolderStatus getFolderStatusNonBlocking(const std::set<AbstractPath>& folderPaths, bool allowUserInteraction, ProcessCallback& procCallback /*throw X*/) { using namespace zen; @@ -49,22 +49,22 @@ FolderStatus getFolderStatusNonBlocking(const std::set<AbstractPath>& folderPath std::vector<ThreadGroup<std::packaged_task<bool()>>> perDeviceThreads; for (const auto& [afsDevice, deviceFolderPaths] : perDevicePaths) { - const size_t parallelOps = getDeviceParallelOps(deviceParallelOps, afsDevice); - - perDeviceThreads.emplace_back(parallelOps, "DirExist: " + utfTo<std::string>(AFS::getDisplayPath(AbstractPath(afsDevice, AfsPath())))); + perDeviceThreads.emplace_back(1, "DirExist: " + utfTo<std::string>(AFS::getDisplayPath(AbstractPath(afsDevice, AfsPath())))); auto& threadGroup = perDeviceThreads.back(); threadGroup.detach(); //don't wait on threads hanging longer than "folderAccessTimeout" + //1. login to network share, connect with Google Drive, etc. + std::shared_future<void> ftAuth = runAsync([afsDevice /*clang bug*/= afsDevice, allowUserInteraction] { AFS::authenticateAccess(afsDevice, allowUserInteraction); /*throw FileError*/ }); + for (const AbstractPath& folderPath : deviceFolderPaths) { - std::packaged_task<bool()> pt([folderPath, allowUserInteraction] + std::packaged_task<bool()> pt([folderPath, ftAuth] { - //1. login to network share, open FTP connection, etc. - AFS::connectNetworkFolder(folderPath, allowUserInteraction); //throw FileError + ftAuth.get(); //throw FileError - //2. check dir existence + /* 2. check dir existence: - /* CAVEAT: the case-sensitive semantics of AFS::itemStillExists() do not fit here! + CAVEAT: the case-sensitive semantics of AFS::itemStillExists() do not fit here! BUT: its implementation happens to be okay for our use: Assume we have a case-insensitive path match: => AFS::itemStillExists() first checks AFS::getItemType() diff --git a/FreeFileSync/Source/base/dir_lock.cpp b/FreeFileSync/Source/base/dir_lock.cpp index b144eb99..d910907b 100755..100644 --- a/FreeFileSync/Source/base/dir_lock.cpp +++ b/FreeFileSync/Source/base/dir_lock.cpp @@ -488,18 +488,5 @@ private: DirLock::DirLock(const Zstring& lockFilePath, const DirLockCallback& notifyStatus, std::chrono::milliseconds cbInterval) //throw FileError { - //#ifdef ZEN_WIN - // const DWORD bufferSize = 10000; - // std::vector<wchar_t> volName(bufferSize); - // if (::GetVolumePathName(lockFilePath.c_str(), //__in LPCTSTR lpszFileName, - // &volName[0], //__out LPTSTR lpszVolumePathName, - // bufferSize)) //__in DWORD cchBufferLength - // { - // const DWORD dt = ::GetDriveType(&volName[0]); - // if (dt == DRIVE_CDROM) - // return; //we don't need a lock for a CD ROM - // } - //#endif -> still relevant? better save the file I/O for the network scenario - sharedLock_ = LockAdmin::instance().retrieve(lockFilePath, notifyStatus, cbInterval); //throw FileError } diff --git a/FreeFileSync/Source/base/dir_lock.h b/FreeFileSync/Source/base/dir_lock.h index 20795804..20795804 100755..100644 --- a/FreeFileSync/Source/base/dir_lock.h +++ b/FreeFileSync/Source/base/dir_lock.h diff --git a/FreeFileSync/Source/base/fatal_error.h b/FreeFileSync/Source/base/fatal_error.h index a27e423b..a27e423b 100755..100644 --- a/FreeFileSync/Source/base/fatal_error.h +++ b/FreeFileSync/Source/base/fatal_error.h diff --git a/FreeFileSync/Source/base/ffs_paths.cpp b/FreeFileSync/Source/base/ffs_paths.cpp index fc993c78..25d8dd58 100755..100644 --- a/FreeFileSync/Source/base/ffs_paths.cpp +++ b/FreeFileSync/Source/base/ffs_paths.cpp @@ -47,13 +47,6 @@ VolumeId fff::getVolumeSerialFfs() //throw FileError bool fff::isPortableVersion() { return false; //users want local installation type: https://freefilesync.org/forum/viewtopic.php?t=5750 - //try - //{ - // return getVolumeSerialFfs() != getVolumeSerialOs(); //throw FileError - //} - //catch (FileError&) {} - //assert(false); - //return false; } diff --git a/FreeFileSync/Source/base/ffs_paths.h b/FreeFileSync/Source/base/ffs_paths.h index 4be75986..4be75986 100755..100644 --- a/FreeFileSync/Source/base/ffs_paths.h +++ b/FreeFileSync/Source/base/ffs_paths.h diff --git a/FreeFileSync/Source/base/file_hierarchy.cpp b/FreeFileSync/Source/base/file_hierarchy.cpp index 39af1e8e..39af1e8e 100755..100644 --- a/FreeFileSync/Source/base/file_hierarchy.cpp +++ b/FreeFileSync/Source/base/file_hierarchy.cpp diff --git a/FreeFileSync/Source/base/file_hierarchy.h b/FreeFileSync/Source/base/file_hierarchy.h index d90c88c4..d90c88c4 100755..100644 --- a/FreeFileSync/Source/base/file_hierarchy.h +++ b/FreeFileSync/Source/base/file_hierarchy.h diff --git a/FreeFileSync/Source/base/generate_logfile.cpp b/FreeFileSync/Source/base/generate_logfile.cpp index 3ce4fbf2..3ce4fbf2 100755..100644 --- a/FreeFileSync/Source/base/generate_logfile.cpp +++ b/FreeFileSync/Source/base/generate_logfile.cpp diff --git a/FreeFileSync/Source/base/generate_logfile.h b/FreeFileSync/Source/base/generate_logfile.h index 9892fa83..9892fa83 100755..100644 --- a/FreeFileSync/Source/base/generate_logfile.h +++ b/FreeFileSync/Source/base/generate_logfile.h diff --git a/FreeFileSync/Source/base/help_provider.h b/FreeFileSync/Source/base/help_provider.h index 8cee813b..8cee813b 100755..100644 --- a/FreeFileSync/Source/base/help_provider.h +++ b/FreeFileSync/Source/base/help_provider.h diff --git a/FreeFileSync/Source/base/icon_buffer.cpp b/FreeFileSync/Source/base/icon_buffer.cpp index e10d25ab..e10d25ab 100755..100644 --- a/FreeFileSync/Source/base/icon_buffer.cpp +++ b/FreeFileSync/Source/base/icon_buffer.cpp diff --git a/FreeFileSync/Source/base/icon_buffer.h b/FreeFileSync/Source/base/icon_buffer.h index 2f5e4e60..2f5e4e60 100755..100644 --- a/FreeFileSync/Source/base/icon_buffer.h +++ b/FreeFileSync/Source/base/icon_buffer.h diff --git a/FreeFileSync/Source/base/icon_loader.cpp b/FreeFileSync/Source/base/icon_loader.cpp index 71b783a2..71b783a2 100755..100644 --- a/FreeFileSync/Source/base/icon_loader.cpp +++ b/FreeFileSync/Source/base/icon_loader.cpp diff --git a/FreeFileSync/Source/base/icon_loader.h b/FreeFileSync/Source/base/icon_loader.h index 8f6b31b9..8f6b31b9 100755..100644 --- a/FreeFileSync/Source/base/icon_loader.h +++ b/FreeFileSync/Source/base/icon_loader.h diff --git a/FreeFileSync/Source/base/localization.cpp b/FreeFileSync/Source/base/localization.cpp index 52ed7fdf..52ed7fdf 100755..100644 --- a/FreeFileSync/Source/base/localization.cpp +++ b/FreeFileSync/Source/base/localization.cpp diff --git a/FreeFileSync/Source/base/localization.h b/FreeFileSync/Source/base/localization.h index bf2f6b60..bf2f6b60 100755..100644 --- a/FreeFileSync/Source/base/localization.h +++ b/FreeFileSync/Source/base/localization.h diff --git a/FreeFileSync/Source/base/lock_holder.h b/FreeFileSync/Source/base/lock_holder.h index 7bc470ba..7bc470ba 100755..100644 --- a/FreeFileSync/Source/base/lock_holder.h +++ b/FreeFileSync/Source/base/lock_holder.h diff --git a/FreeFileSync/Source/base/norm_filter.h b/FreeFileSync/Source/base/norm_filter.h index f96a0aef..f96a0aef 100755..100644 --- a/FreeFileSync/Source/base/norm_filter.h +++ b/FreeFileSync/Source/base/norm_filter.h diff --git a/FreeFileSync/Source/base/parallel_scan.cpp b/FreeFileSync/Source/base/parallel_scan.cpp index a535b6b0..e52749b4 100755..100644 --- a/FreeFileSync/Source/base/parallel_scan.cpp +++ b/FreeFileSync/Source/base/parallel_scan.cpp @@ -10,8 +10,6 @@ #include <zen/basic_math.h> #include <zen/thread.h> #include <zen/scope_guard.h> -//#include "db_file.h" -//#include "lock_holder.h" using namespace zen; using namespace fff; @@ -19,141 +17,29 @@ using namespace fff; namespace { -/* -#ifdef ZEN_WIN - -struct DiskInfo -{ - DiskInfo() : - driveType(DRIVE_UNKNOWN), - diskID(-1) {} - - UINT driveType; - int diskID; // -1 if id could not be determined, this one is filled if driveType == DRIVE_FIXED or DRIVE_REMOVABLE; -}; - -inline -bool operator<(const DiskInfo& lhs, const DiskInfo& rhs) -{ - if (lhs.driveType != rhs.driveType) - return lhs.driveType < rhs.driveType; - - if (lhs.diskID < 0 || rhs.diskID < 0) - return false; - //consider "same", reason: one volume may be uniquely associated with one disk, while the other volume is associated to the same disk AND another one! - //volume <-> disk is 0..N:1..N - - return lhs.diskID < rhs.diskID ; -} - - -DiskInfo retrieveDiskInfo(const Zstring& itemPath) -{ - std::vector<wchar_t> volName(std::max(pathName.size(), static_cast<size_t>(10000))); - - DiskInfo output; - - //full pathName need not yet exist! - if (!::GetVolumePathName(itemPath.c_str(), //__in LPCTSTR lpszFileName, - &volName[0], //__out LPTSTR lpszVolumePathName, - static_cast<DWORD>(volName.size()))) //__in DWORD cchBufferLength - return output; - - const Zstring rootPathName = &volName[0]; - - output.driveType = ::GetDriveType(rootPathName.c_str()); - - if (output.driveType == DRIVE_NO_ROOT_DIR) //these two should be the same error category - output.driveType = DRIVE_UNKNOWN; - - if (output.driveType != DRIVE_FIXED && output.driveType != DRIVE_REMOVABLE) - return output; //no reason to get disk ID - - //go and find disk id: - - //format into form: "\\.\C:" - Zstring volnameFmt = rootPathName; - if (endsWith(volnameFmt, FILE_NAME_SEPARATOR)) - volnameFmt.pop_back(); - volnameFmt = L"\\\\.\\" + volnameFmt; - - HANDLE hVolume = ::CreateFile(volnameFmt.c_str(), - 0, - FILE_SHARE_READ | FILE_SHARE_WRITE, - nullptr, - OPEN_EXISTING, - 0, - nullptr); - if (hVolume == INVALID_HANDLE_VALUE) - return output; - ZEN_ON_SCOPE_EXIT(::CloseHandle(hVolume)); - - std::vector<std::byte> buffer(sizeof(VOLUME_DISK_EXTENTS) + sizeof(DISK_EXTENT)); //reserve buffer for at most one disk! call below will then fail if volume spans multiple disks! - - DWORD bytesReturned = 0; - if (!::DeviceIoControl(hVolume, //_In_ HANDLE hDevice, - IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, //_In_ DWORD dwIoControlCode, - nullptr, //_In_opt_ LPVOID lpInBuffer, - 0, //_In_ DWORD nInBufferSize, - &buffer[0], //_Out_opt_ LPVOID lpOutBuffer, - static_cast<DWORD>(buffer.size()), //_In_ DWORD nOutBufferSize, - &bytesReturned, //_Out_opt_ LPDWORD lpBytesReturned - nullptr)) //_Inout_opt_ LPOVERLAPPED lpOverlapped - return output; - - const VOLUME_DISK_EXTENTS& volDisks = *reinterpret_cast<VOLUME_DISK_EXTENTS*>(&buffer[0]); - - if (volDisks.NumberOfDiskExtents != 1) - return output; - - output.diskID = volDisks.Extents[0].DiskNumber; - - return output; -} -#endif -*/ - -/* -PERF NOTE - ---------------------------------------------- -|Test case: Reading from two different disks| ---------------------------------------------- -Windows 7: - 1st(unbuffered) |2nd (OS buffered) - ---------------------------------- -1 Thread: 57s | 8s -2 Threads: 39s | 7s - ---------------------------------------------------- -|Test case: Reading two directories from same disk| ---------------------------------------------------- -Windows 7: Windows XP: - 1st(unbuffered) |2nd (OS buffered) 1st(unbuffered) |2nd (OS buffered) - ---------------------------------- ---------------------------------- -1 Thread: 41s | 13s 1 Thread: 45s | 13s -2 Threads: 42s | 11s 2 Threads: 38s | 8s - -=> Traversing does not take any advantage of file locality so that even multiple threads operating on the same disk impose no performance overhead! (even faster on XP) - -std::vector<std::set<DirectoryKey>> separateByDistinctDisk(const std::set<DirectoryKey>& dirkeys) -{ - //use one thread per physical disk: - using DiskKeyMapping = std::map<DiskInfo, std::set<DirectoryKey>>; - DiskKeyMapping tmp; - std::for_each(dirkeys.begin(), dirkeys.end(), - [&](const DirectoryKey& key) { tmp[retrieveDiskInfo(key.dirpathFull_)].insert(key); }); - - std::vector<std::set<DirectoryKey>> buckets; - std::transform(tmp.begin(), tmp.end(), std::back_inserter(buckets), - [&](const DiskKeyMapping::value_type& diskToKey) { return diskToKey.second; }); - return buckets; -} -*/ - -//------------------------------------------------------------------------------------------ - -class AsyncCallback //actor pattern +/* PERF NOTE + + --------------------------------------------- + |Test case: Reading from two different disks| + --------------------------------------------- + Windows 7: + 1st(unbuffered) |2nd (OS buffered) + ---------------------------------- + 1 Thread: 57s | 8s + 2 Threads: 39s | 7s + + --------------------------------------------------- + |Test case: Reading two directories from same disk| + --------------------------------------------------- + Windows 7: Windows XP: + 1st(unbuffered) |2nd (OS buffered) 1st(unbuffered) |2nd (OS buffered) + ---------------------------------- ---------------------------------- + 1 Thread: 41s | 13s 1 Thread: 45s | 13s + 2 Threads: 42s | 11s 2 Threads: 38s | 8s + + => Traversing does not take any advantage of file locality so that even multiple threads operating on the same disk impose no performance overhead! (even faster on XP) */ + +class AsyncCallback { public: AsyncCallback(size_t threadsToFinish, std::chrono::milliseconds cbInterval) : threadsToFinish_(threadsToFinish), cbInterval_(cbInterval) {} @@ -277,10 +163,7 @@ private: std::wstring filePath; { std::lock_guard dummy(lockCurrentStatus_); - - for (const auto& [threadIdx, parallelOps] : activeThreadIdxs_) - parallelOpsTotal += parallelOps; - + parallelOpsTotal = activeThreadIdxs_.size(); filePath = currentFile_; } if (parallelOpsTotal >= 2) @@ -388,15 +271,15 @@ void DirCallback::onFile(const AFS::FileInfo& fi) //throw ThreadInterruption { interruptionPoint(); //throw ThreadInterruption - const Zstring fileRelPath = parentRelPathPf_ + fi.itemName; + const Zstring& relPath = parentRelPathPf_ + fi.itemName; //update status information no matter whether item is excluded or not! if (cfg_.acb.mayReportCurrentFile(cfg_.threadIdx, cfg_.lastReportTime)) - cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, fileRelPath))); + cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, relPath))); //------------------------------------------------------------------------------------ //apply filter before processing (use relative name!) - if (!cfg_.filter.ref().passFileFilter(fileRelPath)) + if (!cfg_.filter.ref().passFileFilter(relPath)) return; //sync.ffs_db database and lock files are excluded via filter! @@ -422,16 +305,16 @@ std::shared_ptr<AFS::TraverserCallback> DirCallback::onFolder(const AFS::FolderI { interruptionPoint(); //throw ThreadInterruption - const Zstring& folderRelPath = parentRelPathPf_ + fi.itemName; + const Zstring& relPath = parentRelPathPf_ + fi.itemName; //update status information no matter whether item is excluded or not! if (cfg_.acb.mayReportCurrentFile(cfg_.threadIdx, cfg_.lastReportTime)) - cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, folderRelPath))); + cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, relPath))); //------------------------------------------------------------------------------------ //apply filter before processing (use relative name!) bool childItemMightMatch = true; - const bool passFilter = cfg_.filter.ref().passDirFilter(folderRelPath, &childItemMightMatch); + const bool passFilter = cfg_.filter.ref().passDirFilter(relPath, &childItemMightMatch); if (!passFilter && !childItemMightMatch) return nullptr; //do NOT traverse subdirs //else: attention! ensure directory filtering is applied later to exclude actually filtered directories @@ -444,7 +327,7 @@ std::shared_ptr<AFS::TraverserCallback> DirCallback::onFolder(const AFS::FolderI if (level_ > 100) //Win32 traverser: stack overflow approximately at level 1000 //check after FolderContainer::addSubFolder() for (size_t retryNumber = 0;; ++retryNumber) - switch (reportItemError(replaceCpy(_("Cannot read directory %x."), L"%x", AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, folderRelPath))) + + switch (reportItemError(replaceCpy(_("Cannot read directory %x."), L"%x", AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, relPath))) + L"\n\n" L"Endless recursion.", retryNumber, fi.itemName)) //throw ThreadInterruption { case AbstractFileSystem::TraverserCallback::ON_ERROR_RETRY: @@ -453,7 +336,7 @@ std::shared_ptr<AFS::TraverserCallback> DirCallback::onFolder(const AFS::FolderI return nullptr; } - return std::make_shared<DirCallback>(cfg_, folderRelPath + FILE_NAME_SEPARATOR, subFolder, level_ + 1); + return std::make_shared<DirCallback>(cfg_, relPath + FILE_NAME_SEPARATOR, subFolder, level_ + 1); } @@ -461,11 +344,11 @@ DirCallback::HandleLink DirCallback::onSymlink(const AFS::SymlinkInfo& si) //thr { interruptionPoint(); //throw ThreadInterruption - const Zstring& linkRelPath = parentRelPathPf_ + si.itemName; + const Zstring& relPath = parentRelPathPf_ + si.itemName; //update status information no matter whether item is excluded or not! if (cfg_.acb.mayReportCurrentFile(cfg_.threadIdx, cfg_.lastReportTime)) - cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, linkRelPath))); + cfg_.acb.reportCurrentFile(AFS::getDisplayPath(AFS::appendRelPath(cfg_.baseFolderPath, relPath))); switch (cfg_.handleSymlinks) { @@ -473,7 +356,7 @@ DirCallback::HandleLink DirCallback::onSymlink(const AFS::SymlinkInfo& si) //thr return LINK_SKIP; case SymLinkHandling::DIRECT: - if (cfg_.filter.ref().passFileFilter(linkRelPath)) //always use file filter: Link type may not be "stable" on Linux! + if (cfg_.filter.ref().passFileFilter(relPath)) //always use file filter: Link type may not be "stable" on Linux! { output_.addSubLink(si.itemName, LinkAttributes(si.modTime)); cfg_.acb.incItemsScanned(); //add 1 element to the progress indicator @@ -483,10 +366,10 @@ DirCallback::HandleLink DirCallback::onSymlink(const AFS::SymlinkInfo& si) //thr case SymLinkHandling::FOLLOW: //filter symlinks before trying to follow them: handle user-excluded broken symlinks! //since we don't know yet what type the symlink will resolve to, only do this when both filter variants agree: - if (!cfg_.filter.ref().passFileFilter(linkRelPath)) + if (!cfg_.filter.ref().passFileFilter(relPath)) { bool childItemMightMatch = true; - if (!cfg_.filter.ref().passDirFilter(linkRelPath, &childItemMightMatch)) + if (!cfg_.filter.ref().passDirFilter(relPath, &childItemMightMatch)) if (!childItemMightMatch) return LINK_SKIP; } @@ -520,7 +403,6 @@ DirCallback::HandleError DirCallback::reportError(const std::wstring& msg, size_ void fff::parallelDeviceTraversal(const std::set<DirectoryKey>& foldersToRead, std::map<DirectoryKey, DirectoryValue>& output, - const std::map<AfsDevice, size_t>& deviceParallelOps, const TravErrorCb& onError, const TravStatusCb& onStatusUpdate, std::chrono::milliseconds cbInterval) { @@ -546,14 +428,13 @@ void fff::parallelDeviceTraversal(const std::set<DirectoryKey>& foldersToRead, for (const auto& [afsDevice, dirKeys] : perDeviceFolders) { const int threadIdx = static_cast<int>(worker.size()); - const size_t parallelOps = getDeviceParallelOps(deviceParallelOps, afsDevice); - + const size_t parallelOps = 1; std::map<DirectoryKey, DirectoryValue*> workload; for (const DirectoryKey& key : dirKeys) workload.emplace(key, &output[key]); //=> DirectoryValue* unshared for lock-free worker-thread access - worker.emplace_back([afsDevice = afsDevice /*clang bug :>*/, workload, threadIdx, &acb, parallelOps]() mutable + worker.emplace_back([afsDevice /*clang bug*/= afsDevice, workload, threadIdx, &acb, parallelOps]() mutable { setCurrentThreadName(("Comp Worker[" + numberTo<std::string>(threadIdx) + "]").c_str()); diff --git a/FreeFileSync/Source/base/parallel_scan.h b/FreeFileSync/Source/base/parallel_scan.h index 71dfd586..9aa4bc82 100755..100644 --- a/FreeFileSync/Source/base/parallel_scan.h +++ b/FreeFileSync/Source/base/parallel_scan.h @@ -59,7 +59,6 @@ using TravStatusCb = std::function< void (const std void parallelDeviceTraversal(const std::set<DirectoryKey>& foldersToRead, std::map<DirectoryKey, DirectoryValue>& output, - const std::map<AfsDevice, size_t>& deviceParallelOps, const TravErrorCb& onError, const TravStatusCb& onStatusUpdate, //NOT optional std::chrono::milliseconds cbInterval); } diff --git a/FreeFileSync/Source/base/parse_lng.h b/FreeFileSync/Source/base/parse_lng.h index e1e5e57d..e1e5e57d 100755..100644 --- a/FreeFileSync/Source/base/parse_lng.h +++ b/FreeFileSync/Source/base/parse_lng.h diff --git a/FreeFileSync/Source/base/parse_plural.h b/FreeFileSync/Source/base/parse_plural.h index d80a0dca..d80a0dca 100755..100644 --- a/FreeFileSync/Source/base/parse_plural.h +++ b/FreeFileSync/Source/base/parse_plural.h diff --git a/FreeFileSync/Source/base/path_filter.cpp b/FreeFileSync/Source/base/path_filter.cpp index f1fa1580..f1fa1580 100755..100644 --- a/FreeFileSync/Source/base/path_filter.cpp +++ b/FreeFileSync/Source/base/path_filter.cpp diff --git a/FreeFileSync/Source/base/path_filter.h b/FreeFileSync/Source/base/path_filter.h index 29705b06..29705b06 100755..100644 --- a/FreeFileSync/Source/base/path_filter.h +++ b/FreeFileSync/Source/base/path_filter.h diff --git a/FreeFileSync/Source/base/perf_check.cpp b/FreeFileSync/Source/base/perf_check.cpp index 9493a8a2..9493a8a2 100755..100644 --- a/FreeFileSync/Source/base/perf_check.cpp +++ b/FreeFileSync/Source/base/perf_check.cpp diff --git a/FreeFileSync/Source/base/perf_check.h b/FreeFileSync/Source/base/perf_check.h index 2e9ccc6d..2e9ccc6d 100755..100644 --- a/FreeFileSync/Source/base/perf_check.h +++ b/FreeFileSync/Source/base/perf_check.h diff --git a/FreeFileSync/Source/base/process_callback.h b/FreeFileSync/Source/base/process_callback.h index 34c4c7e5..34c4c7e5 100755..100644 --- a/FreeFileSync/Source/base/process_callback.h +++ b/FreeFileSync/Source/base/process_callback.h diff --git a/FreeFileSync/Source/base/process_xml.cpp b/FreeFileSync/Source/base/process_xml.cpp index 227fada4..56e192ac 100755..100644 --- a/FreeFileSync/Source/base/process_xml.cpp +++ b/FreeFileSync/Source/base/process_xml.cpp @@ -22,7 +22,7 @@ using namespace fff; //functionally needed for correct overload resolution!!! namespace { //------------------------------------------------------------------------------------------------------------------------------- -const int XML_FORMAT_VER_GLOBAL = 11; //2018-09-09 +const int XML_FORMAT_VER_GLOBAL = 12; //2019-02-09 const int XML_FORMAT_VER_FFS_CFG = 14; //2018-08-13 //------------------------------------------------------------------------------------------------------------------------------- } @@ -1414,24 +1414,27 @@ void readConfig(const XmlIn& in, XmlGlobalSettings& cfg, int formatVer) XmlIn inOpt = inGeneral["OptionalDialogs"]; inOpt["ConfirmStartSync" ].attribute("Enabled", cfg.confirmDlgs.confirmSyncStart); inOpt["ConfirmSaveConfig" ].attribute("Enabled", cfg.confirmDlgs.popupOnConfigChange); - inOpt["ConfirmExternalCommandMassInvoke"].attribute("Enabled", cfg.confirmDlgs.confirmExternalCommandMassInvoke); - inOpt["WarnUnresolvedConflicts" ].attribute("Enabled", cfg.warnDlgs.warnUnresolvedConflicts); - inOpt["WarnNotEnoughDiskSpace" ].attribute("Enabled", cfg.warnDlgs.warnNotEnoughDiskSpace); - inOpt["WarnSignificantDifference" ].attribute("Enabled", cfg.warnDlgs.warnSignificantDifference); - inOpt["WarnRecycleBinNotAvailable" ].attribute("Enabled", cfg.warnDlgs.warnRecyclerMissing); - inOpt["WarnInputFieldEmpty" ].attribute("Enabled", cfg.warnDlgs.warnInputFieldEmpty); - inOpt["WarnModificationTimeError" ].attribute("Enabled", cfg.warnDlgs.warnModificationTimeError); - inOpt["WarnDependentFolderPair" ].attribute("Enabled", cfg.warnDlgs.warnDependentFolderPair); - inOpt["WarnDependentBaseFolders" ].attribute("Enabled", cfg.warnDlgs.warnDependentBaseFolders); - inOpt["WarnDirectoryLockFailed" ].attribute("Enabled", cfg.warnDlgs.warnDirectoryLockFailed); - inOpt["WarnVersioningFolderPartOfSync"].attribute("Enabled", cfg.warnDlgs.warnVersioningFolderPartOfSync); + inOpt["ConfirmExternalCommandMassInvoke"].attribute("Enabled", cfg.confirmDlgs.confirmCommandMassInvoke); + inOpt["WarnUnresolvedConflicts" ].attribute("Enabled", cfg.warnDlgs.warnUnresolvedConflicts); + inOpt["WarnNotEnoughDiskSpace" ].attribute("Enabled", cfg.warnDlgs.warnNotEnoughDiskSpace); + inOpt["WarnSignificantDifference" ].attribute("Enabled", cfg.warnDlgs.warnSignificantDifference); + inOpt["WarnRecycleBinNotAvailable" ].attribute("Enabled", cfg.warnDlgs.warnRecyclerMissing); + inOpt["WarnInputFieldEmpty" ].attribute("Enabled", cfg.warnDlgs.warnInputFieldEmpty); + inOpt["WarnModificationTimeError" ].attribute("Enabled", cfg.warnDlgs.warnModificationTimeError); + inOpt["WarnDependentFolderPair" ].attribute("Enabled", cfg.warnDlgs.warnDependentFolderPair); + inOpt["WarnDependentBaseFolders" ].attribute("Enabled", cfg.warnDlgs.warnDependentBaseFolders); + inOpt["WarnDirectoryLockFailed" ].attribute("Enabled", cfg.warnDlgs.warnDirectoryLockFailed); + inOpt["WarnVersioningFolderPartOfSync" ].attribute("Enabled", cfg.warnDlgs.warnVersioningFolderPartOfSync); } else { XmlIn inOpt = inGeneral["OptionalDialogs"]; inOpt["ConfirmStartSync" ].attribute("Show", cfg.confirmDlgs.confirmSyncStart); inOpt["ConfirmSaveConfig" ].attribute("Show", cfg.confirmDlgs.popupOnConfigChange); - inOpt["ConfirmExternalCommandMassInvoke"].attribute("Show", cfg.confirmDlgs.confirmExternalCommandMassInvoke); + if (formatVer < 12) //TODO: remove old parameter after migration! 2019-02-09 + inOpt["ConfirmExternalCommandMassInvoke"].attribute("Show", cfg.confirmDlgs.confirmCommandMassInvoke); + else + inOpt["ConfirmCommandMassInvoke"].attribute("Show", cfg.confirmDlgs.confirmCommandMassInvoke); inOpt["WarnFolderNotExisting" ].attribute("Show", cfg.warnDlgs.warnFolderNotExisting); inOpt["WarnFoldersDifferInCase" ].attribute("Show", cfg.warnDlgs.warnFoldersDifferInCase); inOpt["WarnUnresolvedConflicts" ].attribute("Show", cfg.warnDlgs.warnUnresolvedConflicts); @@ -1446,7 +1449,7 @@ void readConfig(const XmlIn& in, XmlGlobalSettings& cfg, int formatVer) inOpt["WarnVersioningFolderPartOfSync"].attribute("Show", cfg.warnDlgs.warnVersioningFolderPartOfSync); } - //gui specific global settings (optional) + //GUI-specific global settings (optional) XmlIn inGui = in["Gui"]; XmlIn inWnd = inGui["MainDialog"]; @@ -2049,9 +2052,9 @@ void writeConfig(const XmlGlobalSettings& cfg, XmlOut& out) outGeneral["ProgressDialog" ].attribute("AutoClose", cfg.autoCloseProgressDialog); XmlOut outOpt = outGeneral["OptionalDialogs"]; - outOpt["ConfirmStartSync" ].attribute("Show", cfg.confirmDlgs.confirmSyncStart); - outOpt["ConfirmSaveConfig" ].attribute("Show", cfg.confirmDlgs.popupOnConfigChange); - outOpt["ConfirmExternalCommandMassInvoke"].attribute("Show", cfg.confirmDlgs.confirmExternalCommandMassInvoke); + outOpt["ConfirmStartSync" ].attribute("Show", cfg.confirmDlgs.confirmSyncStart); + outOpt["ConfirmSaveConfig" ].attribute("Show", cfg.confirmDlgs.popupOnConfigChange); + outOpt["ConfirmCommandMassInvoke" ].attribute("Show", cfg.confirmDlgs.confirmCommandMassInvoke); outOpt["WarnFolderNotExisting" ].attribute("Show", cfg.warnDlgs.warnFolderNotExisting); outOpt["WarnFoldersDifferInCase" ].attribute("Show", cfg.warnDlgs.warnFoldersDifferInCase); outOpt["WarnUnresolvedConflicts" ].attribute("Show", cfg.warnDlgs.warnUnresolvedConflicts); diff --git a/FreeFileSync/Source/base/process_xml.h b/FreeFileSync/Source/base/process_xml.h index fe4687ba..bdda02f1 100755..100644 --- a/FreeFileSync/Source/base/process_xml.h +++ b/FreeFileSync/Source/base/process_xml.h @@ -85,13 +85,13 @@ struct ConfirmationDialogs { bool popupOnConfigChange = true; bool confirmSyncStart = true; - bool confirmExternalCommandMassInvoke = true; + bool confirmCommandMassInvoke = true; }; inline bool operator==(const ConfirmationDialogs& lhs, const ConfirmationDialogs& rhs) { return lhs.popupOnConfigChange == rhs.popupOnConfigChange && lhs.confirmSyncStart == rhs.confirmSyncStart && - lhs.confirmExternalCommandMassInvoke == rhs.confirmExternalCommandMassInvoke; + lhs.confirmCommandMassInvoke == rhs.confirmCommandMassInvoke; } inline bool operator!=(const ConfirmationDialogs& lhs, const ConfirmationDialogs& rhs) { return !(lhs == rhs); } diff --git a/FreeFileSync/Source/base/resolve_path.cpp b/FreeFileSync/Source/base/resolve_path.cpp index b43d3463..5acf4355 100755..100644 --- a/FreeFileSync/Source/base/resolve_path.cpp +++ b/FreeFileSync/Source/base/resolve_path.cpp @@ -156,16 +156,16 @@ Zstring expandVolumeName(Zstring pathPhrase) // [volname]:\folder [volnam trim(pathPhrase, true, false); if (startsWith(pathPhrase, Zstr("["))) { - size_t posEnd = pathPhrase.find(Zstr("]")); + const size_t posEnd = pathPhrase.find(Zstr("]")); if (posEnd != Zstring::npos) { Zstring volName = Zstring(pathPhrase.c_str() + 1, posEnd - 1); Zstring relPath = Zstring(pathPhrase.c_str() + posEnd + 1); - if (startsWith(relPath, Zstr(':'))) - relPath = afterFirst(relPath, Zstr(':'), IF_MISSING_RETURN_NONE); if (startsWith(relPath, FILE_NAME_SEPARATOR)) relPath = afterFirst(relPath, FILE_NAME_SEPARATOR, IF_MISSING_RETURN_NONE); + else if (startsWith(relPath, Zstr(":\\"))) //Win-only + relPath = afterFirst(relPath, Zstr('\\'), IF_MISSING_RETURN_NONE); return "/.../[" + volName + "]/" + relPath; } } diff --git a/FreeFileSync/Source/base/resolve_path.h b/FreeFileSync/Source/base/resolve_path.h index 8b8f131e..8b8f131e 100755..100644 --- a/FreeFileSync/Source/base/resolve_path.h +++ b/FreeFileSync/Source/base/resolve_path.h diff --git a/FreeFileSync/Source/base/return_codes.h b/FreeFileSync/Source/base/return_codes.h index 9bbd2b11..9bbd2b11 100755..100644 --- a/FreeFileSync/Source/base/return_codes.h +++ b/FreeFileSync/Source/base/return_codes.h diff --git a/FreeFileSync/Source/base/soft_filter.h b/FreeFileSync/Source/base/soft_filter.h index d0552c2d..d0552c2d 100755..100644 --- a/FreeFileSync/Source/base/soft_filter.h +++ b/FreeFileSync/Source/base/soft_filter.h diff --git a/FreeFileSync/Source/base/status_handler.cpp b/FreeFileSync/Source/base/status_handler.cpp index aba4810c..aba4810c 100755..100644 --- a/FreeFileSync/Source/base/status_handler.cpp +++ b/FreeFileSync/Source/base/status_handler.cpp diff --git a/FreeFileSync/Source/base/status_handler.h b/FreeFileSync/Source/base/status_handler.h index 59d5c80c..59d5c80c 100755..100644 --- a/FreeFileSync/Source/base/status_handler.h +++ b/FreeFileSync/Source/base/status_handler.h diff --git a/FreeFileSync/Source/base/status_handler_impl.h b/FreeFileSync/Source/base/status_handler_impl.h index 0153f748..ba7d588a 100755..100644 --- a/FreeFileSync/Source/base/status_handler_impl.h +++ b/FreeFileSync/Source/base/status_handler_impl.h @@ -246,8 +246,7 @@ private: std::lock_guard dummy(lockCurrentStatus_); for (const auto& sbp : statusByPriority_) - parallelOpsTotal += sbp.size(); - + parallelOpsTotal += sbp.empty() ? 0 : 1; statusMsg = [&] { for (const std::vector<ThreadStatus>& sbp : statusByPriority_) @@ -390,7 +389,6 @@ struct ParallelContext namespace { void massParallelExecute(const std::vector<std::pair<AbstractPath, ParallelWorkItem>>& workload, - const std::map<AfsDevice, size_t>& deviceParallelOps, const std::string& threadGroupName, ProcessCallback& callback /*throw X*/) { @@ -422,10 +420,9 @@ void massParallelExecute(const std::vector<std::pair<AbstractPath, ParallelWorkI //Attention: carefully orchestrate access to deviceThreadGroups and its contained worker threads! e.g. synchronize potential access during ~DeviceThreadGroup! for (const auto& [afsDevice, wl] : perDeviceWorkload) { - const size_t parallelOps = getDeviceParallelOps(deviceParallelOps, afsDevice); const size_t statusPrio = deviceThreadGroups.size(); - auto scheduleExtraTask = [&acb, &deviceThreadGroupsShared, afsDevice = afsDevice /*clang bug :>*/](const AfsPath& afsPath, const ParallelWorkItem& task) + auto scheduleExtraTask = [&acb, &deviceThreadGroupsShared, afsDevice /*clang bug*/= afsDevice](const AfsPath& afsPath, const ParallelWorkItem& task) { const AbstractPath itemPath(afsDevice, afsPath); @@ -446,10 +443,11 @@ void massParallelExecute(const std::vector<std::pair<AbstractPath, ParallelWorkI }); }); }; - deviceThreadGroups.emplace(afsDevice, ThreadGroupContext(parallelOps, - threadGroupName + " " + utfTo<std::string>(AFS::getDisplayPath(AbstractPath(afsDevice, AfsPath()))), - statusPrio, - scheduleExtraTask)); + deviceThreadGroups.emplace(afsDevice, ThreadGroupContext( + 1, + threadGroupName + " " + utfTo<std::string>(AFS::getDisplayPath(AbstractPath(afsDevice, AfsPath()))), + statusPrio, + scheduleExtraTask)); } deviceThreadGroupsShared.access([&](auto*& deviceThreadGroupsPtr) { deviceThreadGroupsPtr = &deviceThreadGroups; }); //[!] deviceThreadGroups is shared with worker threads from here on! diff --git a/FreeFileSync/Source/base/structures.cpp b/FreeFileSync/Source/base/structures.cpp index ea5f9ad3..c900206a 100755..100644 --- a/FreeFileSync/Source/base/structures.cpp +++ b/FreeFileSync/Source/base/structures.cpp @@ -323,28 +323,6 @@ std::wstring fff::getSymbol(SyncOperation op) namespace { -/* -int daysSinceBeginOfWeek(int dayOfWeek) //0-6, 0=Monday, 6=Sunday -{ - assert(0 <= dayOfWeek && dayOfWeek <= 6); -#ifdef ZEN_WIN - DWORD firstDayOfWeek = 0; - if (::GetLocaleInfo(LOCALE_USER_DEFAULT, //__in LCID Locale, - LOCALE_IFIRSTDAYOFWEEK | // first day of week specifier, 0-6, 0=Monday, 6=Sunday - LOCALE_RETURN_NUMBER, //__in LCTYPE LCType, - reinterpret_cast<LPTSTR>(&firstDayOfWeek), //__out LPTSTR lpLCData, - sizeof(firstDayOfWeek) / sizeof(TCHAR)) > 0) //__in int cchData - { - assert(firstDayOfWeek <= 6); - return (dayOfWeek + (7 - firstDayOfWeek)) % 7; - } - else //default -#endif - return dayOfWeek; //let all weeks begin with monday -} -*/ - - time_t resolve(size_t value, UnitTime unit, time_t defaultVal) { TimeComp tcLocal = getLocalTime(); @@ -360,19 +338,6 @@ time_t resolve(size_t value, UnitTime unit, time_t defaultVal) tcLocal.hour = 0; //0-23 return localToTimeT(tcLocal); //convert local time back to UTC - //case UnitTime::THIS_WEEK: - //{ - // localTimeFmt->tm_sec = 0; //0-61 - // localTimeFmt->tm_min = 0; //0-59 - // localTimeFmt->tm_hour = 0; //0-23 - // const time_t timeFrom = ::mktime(localTimeFmt); - - // int dayOfWeek = (localTimeFmt->tm_wday + 6) % 7; //tm_wday := days since Sunday 0-6 - // // +6 == -1 in Z_7 - - // return int64_t(timeFrom) - daysSinceBeginOfWeek(dayOfWeek) * 24 * 3600; - //} - case UnitTime::THIS_MONTH: tcLocal.second = 0; //0-61 tcLocal.minute = 0; //0-59 diff --git a/FreeFileSync/Source/base/structures.h b/FreeFileSync/Source/base/structures.h index 372d48b5..372d48b5 100755..100644 --- a/FreeFileSync/Source/base/structures.h +++ b/FreeFileSync/Source/base/structures.h diff --git a/FreeFileSync/Source/base/synchronization.cpp b/FreeFileSync/Source/base/synchronization.cpp index 8b86503b..ce38944e 100755..100644 --- a/FreeFileSync/Source/base/synchronization.cpp +++ b/FreeFileSync/Source/base/synchronization.cpp @@ -489,12 +489,12 @@ std::optional<AFS::ItemType> itemStillExists(const AbstractPath& ap, std::mutex& { return parallelScope([ap] { return AFS::itemStillExists(ap); /*throw FileError*/ }, singleThread); } inline -bool removeFileIfExists(const AbstractPath& ap, std::mutex& singleThread) //throw FileError -{ return parallelScope([ap] { return AFS::removeFileIfExists(ap); /*throw FileError*/ }, singleThread); } +void removeFileIfExists(const AbstractPath& ap, std::mutex& singleThread) //throw FileError +{ parallelScope([ap] { AFS::removeFileIfExists(ap); /*throw FileError*/ }, singleThread); } inline -bool removeSymlinkIfExists(const AbstractPath& ap, std::mutex& singleThread) //throw FileError -{ return parallelScope([ap] { return AFS::removeSymlinkIfExists(ap); /*throw FileError*/ }, singleThread); } +void removeSymlinkIfExists(const AbstractPath& ap, std::mutex& singleThread) //throw FileError +{ parallelScope([ap] { AFS::removeSymlinkIfExists(ap); /*throw FileError*/ }, singleThread); } inline void moveAndRenameItem(const AbstractPath& apSource, const AbstractPath& apTarget, std::mutex& singleThread) //throw FileError, ErrorDifferentVolume @@ -732,7 +732,7 @@ void DeletionHandler::removeDirWithCallback(const AbstractPath& folderPath,//thr { statReporter.reportStatus(replaceCpy(statusText, L"%x", fmtPath(displayPath))); //throw ThreadInterruption statReporter.reportDelta(1, 0); //it would be more correct to report *after* work was done! - //OTOH: ThreadInterruption must not happen after last deletion was successful: allow for transactional file model update! + //OTOH: ThreadInterruption must not happen just after last deletion was successful: allow for transactional file model update! warn_static("=> indeed; fix!?") }; static_assert(std::is_const_v<decltype(txtRemovingFile_)>, "callbacks better be thread-safe!"); @@ -926,24 +926,24 @@ public: std::vector<FileError>& errorsModTime; DeletionHandler& delHandlerLeft; DeletionHandler& delHandlerRight; - size_t threadCount; }; static void runSync(SyncCtx& syncCtx, BaseFolderPair& baseFolder, ProcessCallback& cb) { - runPass(PASS_ZERO, syncCtx, baseFolder, cb); //prepare file moves - runPass(PASS_ONE, syncCtx, baseFolder, cb); //delete files (or overwrite big ones with smaller ones) - runPass(PASS_TWO, syncCtx, baseFolder, cb); //copy rest + runPass(PassNo::zero, syncCtx, baseFolder, cb); //prepare file moves + runPass(PassNo::one, syncCtx, baseFolder, cb); //delete files (or overwrite big ones with smaller ones) + runPass(PassNo::two, syncCtx, baseFolder, cb); //copy rest } private: friend class Workload; - enum PassNo + + enum class PassNo { - PASS_ZERO, //prepare file moves - PASS_ONE, //delete files - PASS_TWO, //create, modify - PASS_NEVER //skip item + zero, //prepare file moves + one, //delete files + two, //create, modify + never //skip item }; FolderPairSyncer(SyncCtx& syncCtx, std::mutex& singleThread, AsyncCallback& acb) : @@ -1048,20 +1048,19 @@ Notes: - All threads share a single mutex, unlocked only during file I/O => do N void FolderPairSyncer::runPass(PassNo pass, SyncCtx& syncCtx, BaseFolderPair& baseFolder, ProcessCallback& cb) //throw X { - const size_t threadCount = std::max<size_t>(syncCtx.threadCount, 1); std::mutex singleThread; //only a single worker thread may run at a time, except for parallel file I/O AsyncCallback acb; // FolderPairSyncer fps(syncCtx, singleThread, acb); //manage life time: enclose InterruptibleThread's!!! - Workload workload(threadCount, acb); // + Workload workload(1, acb); workload.addWorkItems(fps.getFolderLevelWorkItems(pass, baseFolder, workload)); //initial workload: set *before* threads get access! std::vector<InterruptibleThread> worker; ZEN_ON_SCOPE_EXIT( for (InterruptibleThread& wt : worker) wt.join (); ); // ZEN_ON_SCOPE_EXIT( for (InterruptibleThread& wt : worker) wt.interrupt(); ); //interrupt all first, then join - for (size_t threadIdx = 0; threadIdx < threadCount; ++threadIdx) + size_t threadIdx = 0; worker.emplace_back([threadIdx, &singleThread, &acb, &workload] { setCurrentThreadName(("Sync Worker[" + numberTo<std::string>(threadIdx) + "]").c_str()); @@ -1109,7 +1108,7 @@ RingBuffer<Workload::WorkItems> FolderPairSyncer::getFolderLevelWorkItems(PassNo //synchronize files: for (FilePair& file : hierObj.refSubFiles()) - if (pass == PASS_ZERO) + if (pass == PassNo::zero) { if (needZeroPass(file)) workItems.push_back([this, &file] { prepareFileMove(file); /*throw ThreadInterruption*/ }); @@ -1249,9 +1248,9 @@ auto FolderPairSyncer::createMoveTargetFolder(FileSystemObject& fsObj) -> CmtfSt if (parallel::itemStillExists(parentFolder->getAbstractPath<sideSrc>(), singleThread_)) //throw FileError { AsyncItemStatReporter statReporter(1, 0, acb_); - try + try { - //target existing: fail/ignore + //target existing: fail/ignore parallel::copyNewFolder(parentFolder->getAbstractPath<sideSrc>(), targetPath, copyFilePermissions_, singleThread_); //throw FileError } catch (FileError&) @@ -1462,34 +1461,34 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const FilePair& file) { case SO_DELETE_LEFT: case SO_DELETE_RIGHT: - return PASS_ONE; + return PassNo::one; case SO_OVERWRITE_LEFT: - return file.getFileSize<LEFT_SIDE>() > file.getFileSize<RIGHT_SIDE>() ? PASS_ONE : PASS_TWO; + return file.getFileSize<LEFT_SIDE>() > file.getFileSize<RIGHT_SIDE>() ? PassNo::one : PassNo::two; case SO_OVERWRITE_RIGHT: - return file.getFileSize<LEFT_SIDE>() < file.getFileSize<RIGHT_SIDE>() ? PASS_ONE : PASS_TWO; + return file.getFileSize<LEFT_SIDE>() < file.getFileSize<RIGHT_SIDE>() ? PassNo::one : PassNo::two; case SO_MOVE_LEFT_FROM: // case SO_MOVE_RIGHT_FROM: // [!] - return PASS_NEVER; + return PassNo::never; case SO_MOVE_LEFT_TO: // case SO_MOVE_RIGHT_TO: //make sure 2-step move is processed in second pass, after move *target* parent directory was created! - return PASS_TWO; + return PassNo::two; case SO_CREATE_NEW_LEFT: case SO_CREATE_NEW_RIGHT: case SO_COPY_METADATA_TO_LEFT: case SO_COPY_METADATA_TO_RIGHT: - return PASS_TWO; + return PassNo::two; case SO_DO_NOTHING: case SO_EQUAL: case SO_UNRESOLVED_CONFLICT: - return PASS_NEVER; + return PassNo::never; } assert(false); - return PASS_NEVER; //dummy + return PassNo::never; //dummy } @@ -1500,7 +1499,7 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const SymlinkPair& link) { case SO_DELETE_LEFT: case SO_DELETE_RIGHT: - return PASS_ONE; //make sure to delete symlinks in first pass, and equally named file or dir in second pass: usecase "overwrite symlink with regular file"! + return PassNo::one; //make sure to delete symlinks in first pass, and equally named file or dir in second pass: usecase "overwrite symlink with regular file"! case SO_OVERWRITE_LEFT: case SO_OVERWRITE_RIGHT: @@ -1508,7 +1507,7 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const SymlinkPair& link) case SO_CREATE_NEW_RIGHT: case SO_COPY_METADATA_TO_LEFT: case SO_COPY_METADATA_TO_RIGHT: - return PASS_TWO; + return PassNo::two; case SO_MOVE_LEFT_FROM: case SO_MOVE_RIGHT_FROM: @@ -1518,10 +1517,10 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const SymlinkPair& link) case SO_DO_NOTHING: case SO_EQUAL: case SO_UNRESOLVED_CONFLICT: - return PASS_NEVER; + return PassNo::never; } assert(false); - return PASS_NEVER; //dummy + return PassNo::never; //dummy } @@ -1532,7 +1531,7 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const FolderPair& folder) { case SO_DELETE_LEFT: case SO_DELETE_RIGHT: - return PASS_ONE; + return PassNo::one; case SO_CREATE_NEW_LEFT: case SO_CREATE_NEW_RIGHT: @@ -1540,7 +1539,7 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const FolderPair& folder) case SO_OVERWRITE_RIGHT: case SO_COPY_METADATA_TO_LEFT: case SO_COPY_METADATA_TO_RIGHT: - return PASS_TWO; + return PassNo::two; case SO_MOVE_LEFT_FROM: case SO_MOVE_RIGHT_FROM: @@ -1550,10 +1549,10 @@ FolderPairSyncer::PassNo FolderPairSyncer::getPass(const FolderPair& folder) case SO_DO_NOTHING: case SO_EQUAL: case SO_UNRESOLVED_CONFLICT: - return PASS_NEVER; + return PassNo::never; } assert(false); - return PASS_NEVER; //dummy + return PassNo::never; //dummy } //--------------------------------------------------------------------------------------------------------------- @@ -1611,14 +1610,18 @@ void FolderPairSyncer::synchronizeFileInt(FilePair& file, SyncOperation syncOp) result.sourceFileId, false, file.isFollowedSymlink<sideSrc>()); } - catch (const FileError& e) + catch (const FileError&) { - bool sourceWasDeleted = false; - try { sourceWasDeleted = !parallel::itemStillExists(file.getAbstractPath<sideSrc>(), singleThread_); /*throw FileError*/ } - catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } //unclear which exception is more relevant + bool sourceExists = true; + try { sourceExists = !!parallel::itemStillExists(file.getAbstractPath<sideSrc>(), singleThread_); /*throw FileError*/ } + catch (const FileError& e2) //more relevant than previous exception (which could be "item not found") + { + throw FileError(replaceCpy(replaceCpy(_("Cannot copy file %x to %y."), + L"%x", L"\n" + fmtPath(AFS::getDisplayPath(file.getAbstractPath<sideSrc>()))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(targetPath))), replaceCpy(e2.toString(), L"\n\n", L"\n")); + } //do not check on type (symlink, file, folder) -> if there is a type change, FFS should not be quiet about it! - - if (sourceWasDeleted) + if (!sourceExists) { statReporter.reportDelta(1, 0); //even if the source item does not exist anymore, significant I/O work was done => report file.removeObject<sideSrc>(); //source deleted meanwhile...nothing was done (logical point of view!) @@ -1830,13 +1833,17 @@ void FolderPairSyncer::synchronizeLinkInt(SymlinkPair& symlink, SyncOperation sy symlink.getLastWriteTime<sideSrc>()); } - catch (const FileError& e) + catch (const FileError&) { bool sourceExists = true; try { sourceExists = !!parallel::itemStillExists(symlink.getAbstractPath<sideSrc>(), singleThread_); /*throw FileError*/ } - catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } //unclear which exception is more relevant + catch (const FileError& e2) //more relevant than previous exception (which could be "item not found") + { + throw FileError(replaceCpy(replaceCpy(_("Cannot copy symbolic link %x to %y."), + L"%x", L"\n" + fmtPath(AFS::getDisplayPath(symlink.getAbstractPath<sideSrc>()))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(targetPath))), replaceCpy(e2.toString(), L"\n\n", L"\n")); + } //do not check on type (symlink, file, folder) -> if there is a type change, FFS should not be quiet about it! - if (!sourceExists) { //even if the source item does not exist anymore, significant I/O work was done => report @@ -2127,7 +2134,7 @@ bool baseFolderDrop(BaseFolderPair& baseFolder, ProcessCallback& callback) { const std::wstring errMsg = tryReportingError([&] { - const FolderStatus status = getFolderStatusNonBlocking({ folderPath }, {} /*deviceParallelOps*/, + const FolderStatus status = getFolderStatusNonBlocking({ folderPath }, false /*allowUserInteraction*/, callback); static_assert(std::is_same_v<decltype(status.failedChecks.begin()->second), FileError>); @@ -2160,7 +2167,7 @@ bool createBaseFolder(BaseFolderPair& baseFolder, bool copyFilePermissions, Proc bool temporaryNetworkDrop = false; const std::wstring errMsg = tryReportingError([&] { - const FolderStatus status = getFolderStatusNonBlocking({ baseFolderPath }, {} /*deviceParallelOps*/, + const FolderStatus status = getFolderStatusNonBlocking({ baseFolderPath }, false /*allowUserInteraction*/, callback); static_assert(std::is_same_v<decltype(status.failedChecks.begin()->second), FileError>); @@ -2217,7 +2224,6 @@ void fff::synchronize(const std::chrono::system_clock::time_point& syncStartTime bool runWithBackgroundPriority, const std::vector<FolderPairSyncCfg>& syncConfig, FolderComparison& folderCmp, - const std::map<AfsDevice, size_t>& deviceParallelOps, WarningDialogs& warnings, ProcessCallback& callback) { @@ -2668,17 +2674,12 @@ void fff::synchronize(const std::chrono::system_clock::time_point& syncStartTime catch (...) { assert(false); } //what is this? ); - size_t parallelOps = std::max(getDeviceParallelOps(deviceParallelOps, baseFolder.getAbstractPath< LEFT_SIDE>().afsDevice), - getDeviceParallelOps(deviceParallelOps, baseFolder.getAbstractPath<RIGHT_SIDE>().afsDevice)); - if (folderPairCfg.handleDeletion == DeletionPolicy::VERSIONING) - parallelOps = std::max(parallelOps, getDeviceParallelOps(deviceParallelOps, versioningFolderPath.afsDevice)); FolderPairSyncer::SyncCtx syncCtx = { verifyCopiedFiles, copyPermissionsFp, failSafeFileCopy, errorsModTime, delHandlerL, delHandlerR, - parallelOps }; FolderPairSyncer::runSync(syncCtx, baseFolder, callback); @@ -2715,7 +2716,8 @@ void fff::synchronize(const std::chrono::system_clock::time_point& syncStartTime //----------------------------------------------------------------------------------------------------- - applyVersioningLimit(versionLimitFolders, deviceParallelOps, callback); //throw X + applyVersioningLimit(versionLimitFolders, + callback); //throw X //------------------- show warnings after end of synchronization -------------------------------------- diff --git a/FreeFileSync/Source/base/synchronization.h b/FreeFileSync/Source/base/synchronization.h index 08d6ca55..b904390f 100755..100644 --- a/FreeFileSync/Source/base/synchronization.h +++ b/FreeFileSync/Source/base/synchronization.h @@ -94,7 +94,6 @@ void synchronize(const std::chrono::system_clock::time_point& syncStartTime, bool runWithBackgroundPriority, const std::vector<FolderPairSyncCfg>& syncConfig, //CONTRACT: syncConfig and folderCmp correspond row-wise! FolderComparison& folderCmp, // - const std::map<AfsDevice, size_t>& deviceParallelOps, WarningDialogs& warnings, ProcessCallback& callback); } diff --git a/FreeFileSync/Source/base/versioning.cpp b/FreeFileSync/Source/base/versioning.cpp index b4499858..f9de8ea4 100755..100644 --- a/FreeFileSync/Source/base/versioning.cpp +++ b/FreeFileSync/Source/base/versioning.cpp @@ -388,7 +388,6 @@ bool fff::operator<(const VersioningLimitFolder& lhs, const VersioningLimitFolde void fff::applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimits, - const std::map<AfsDevice, size_t>& deviceParallelOps, ProcessCallback& callback /*throw X*/) { //--------- determine existing folder paths for traversal --------- @@ -409,7 +408,7 @@ void fff::applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimi //we don't want to show an error if version path does not yet exist! tryReportingError([&] { - const FolderStatus status = getFolderStatusNonBlocking(pathsToCheck, deviceParallelOps, //re-check *all* directories on each try! + const FolderStatus status = getFolderStatusNonBlocking(pathsToCheck, false /*allowUserInteraction*/, callback); //throw X foldersToRead.clear(); for (const AbstractPath& folderPath : status.existing) @@ -456,7 +455,6 @@ void fff::applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimi }; parallelDeviceTraversal(foldersToRead, folderBuf, - deviceParallelOps, onError, onStatusUpdate, //throw X UI_UPDATE_INTERVAL / 2); //every ~50 ms @@ -563,7 +561,7 @@ void fff::applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimi parallelWorkload.emplace_back(folderPath, deleteEmptyFolderTask); for (const auto& [itemPath, isSymlink] : itemsToDelete) - parallelWorkload.emplace_back(itemPath, [isSymlink = isSymlink /*=> clang bug :>*/, &textRemoving, &folderItemCountShared, &deleteEmptyFolderTask](ParallelContext& ctx) //throw ThreadInterruption + parallelWorkload.emplace_back(itemPath, [isSymlink /*clang bug*/= isSymlink, &textRemoving, &folderItemCountShared, &deleteEmptyFolderTask](ParallelContext& ctx) //throw ThreadInterruption { const std::wstring errMsg = tryReportingError([&] //throw ThreadInterruption { @@ -584,8 +582,9 @@ void fff::applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimi assert(parentPath->afsDevice == ctx.itemPath.afsDevice); } - warn_static("get rid of scheduleExtraTask and recursively delete parent folders!? need scheduleExtraTask for something else?") + warn_static("get rid of scheduleExtraTask and just recursively delete parent folders here!? need scheduleExtraTask for something else?") //doable, but call interruptionPoint() for each parent folder }); - massParallelExecute(parallelWorkload, deviceParallelOps, "Versioning Limit", callback /*throw X*/); + massParallelExecute(parallelWorkload, + "Versioning Limit", callback /*throw X*/); } diff --git a/FreeFileSync/Source/base/versioning.h b/FreeFileSync/Source/base/versioning.h index 30bb2fd9..fe6c3a65 100755..100644 --- a/FreeFileSync/Source/base/versioning.h +++ b/FreeFileSync/Source/base/versioning.h @@ -101,7 +101,6 @@ bool operator<(const VersioningLimitFolder& lhs, const VersioningLimitFolder& rh void applyVersioningLimit(const std::set<VersioningLimitFolder>& folderLimits, - const std::map<AfsDevice, size_t>& deviceParallelOps, ProcessCallback& callback /*throw X*/); diff --git a/FreeFileSync/Source/fs/abstract.cpp b/FreeFileSync/Source/fs/abstract.cpp index d247bd4e..294484e3 100755..100644 --- a/FreeFileSync/Source/fs/abstract.cpp +++ b/FreeFileSync/Source/fs/abstract.cpp @@ -108,31 +108,46 @@ AFS::FileCopyResult AFS::copyFileAsStream(const AfsPath& afsPathSource, const St const AbstractPath& apTarget, const IOCallback& notifyUnbufferedIO /*throw X*/) const { int64_t totalUnbufferedIO = 0; + IOCallbackDivider cbd(notifyUnbufferedIO, totalUnbufferedIO); - auto streamIn = getInputStream(afsPathSource, IOCallbackDivider(notifyUnbufferedIO, totalUnbufferedIO)); //throw FileError, ErrorFileLocked + int64_t totalBytesRead = 0; + int64_t totalBytesWritten = 0; + auto notifyUnbufferedRead = [&](int64_t bytesDelta) { totalBytesRead += bytesDelta; cbd(bytesDelta); }; + auto notifyUnbufferedWrite = [&](int64_t bytesDelta) { totalBytesWritten += bytesDelta; cbd(bytesDelta); }; + //-------------------------------------------------------------------------------------------------------- + + auto streamIn = getInputStream(afsPathSource, notifyUnbufferedRead); //throw FileError, ErrorFileLocked StreamAttributes attrSourceNew = {}; //try to get the most current attributes if possible (input file might have changed after comparison!) if (std::optional<StreamAttributes> attr = streamIn->getAttributesBuffered()) //throw FileError - attrSourceNew = *attr; //Native/MTP/Gdrive + attrSourceNew = *attr; //Native/MTP/Google Drive else //use more stale ones: attrSourceNew = attrSource; //SFTP/FTP //TODO: evaluate: consequences of stale attributes //target existing: undefined behavior! (fail/overwrite/auto-rename) - auto streamOut = getOutputStream(apTarget, attrSourceNew.fileSize, attrSourceNew.modTime, IOCallbackDivider(notifyUnbufferedIO, totalUnbufferedIO)); //throw FileError + auto streamOut = getOutputStream(apTarget, attrSourceNew.fileSize, attrSourceNew.modTime, notifyUnbufferedWrite); //throw FileError bufferedStreamCopy(*streamIn, *streamOut); //throw FileError, ErrorFileLocked, X const AFS::FinalizeResult finResult = streamOut->finalize(); //throw FileError, X - //check if "expected == actual number of bytes written" - //-> extra check: bytes reported via notifyUnbufferedIO() should match actual number of bytes written - if (totalUnbufferedIO != 2 * makeSigned(attrSourceNew.fileSize)) + //catch file I/O bugs + read/write conflicts: (note: different check than inside AbstractFileSystem::OutputStream::finalize() => checks notifyUnbufferedIO()!) + ZEN_ON_SCOPE_FAIL(try { removeFilePlain(apTarget); /*throw FileError*/ } + catch (FileError& e) { (void)e; }); //after finalize(): not guarded by ~AFS::OutputStream() anymore! + + if (totalBytesRead != makeSigned(attrSourceNew.fileSize)) throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(getDisplayPath(afsPathSource))), replaceCpy(replaceCpy(_("Unexpected size of data stream.\nExpected: %x bytes\nActual: %y bytes"), - L"%x", numberTo<std::wstring>(2 * attrSourceNew.fileSize)), - L"%y", numberTo<std::wstring>(totalUnbufferedIO)) + L" [notifyUnbufferedIO]"); + L"%x", numberTo<std::wstring>(attrSourceNew.fileSize)), + L"%y", numberTo<std::wstring>(totalBytesRead)) + L" [notifyUnbufferedRead]"); + + if (totalBytesWritten != totalBytesRead) + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getDisplayPath(apTarget))), + replaceCpy(replaceCpy(_("Unexpected size of data stream.\nExpected: %x bytes\nActual: %y bytes"), + L"%x", numberTo<std::wstring>(totalBytesRead)), + L"%y", numberTo<std::wstring>(totalBytesWritten)) + L" [notifyUnbufferedWrite]"); AFS::FileCopyResult cpResult; cpResult.fileSize = attrSourceNew.fileSize; @@ -145,7 +160,7 @@ AFS::FileCopyResult AFS::copyFileAsStream(const AfsPath& afsPathSource, const St - GVFS failing to set modTime for FTP: https://freefilesync.org/forum/viewtopic.php?t=2372 - GVFS failing to set modTime for MTP: https://freefilesync.org/forum/viewtopic.php?t=2803 - MTP failing to set modTime in general: fail non-silently rather than silently during file creation - - FTP failing to set modTime for servers lacking MFMT-support */ + - FTP failing to set modTime for servers without MFMT-support */ return cpResult; } @@ -189,7 +204,7 @@ AFS::FileCopyResult AFS::copyFileTransactional(const AbstractPath& apSource, con Zstring tmpName = beforeLast(fileName, Zstr('.'), IF_MISSING_RETURN_ALL); - //don't make the temp name longer than the original; avoid hitting file system name length limitations: "lpMaximumComponentLength is commonly 255 characters" + //don't make the temp name longer than the original when hitting file system name length limitations: "lpMaximumComponentLength is commonly 255 characters" while (tmpName.size() > 200) //BUT don't trim short names! we want early failure on filename-related issues tmpName = getUnicodeSubstring(tmpName, 0 /*uniPosFirst*/, unicodeLength(tmpName) / 2 /*uniPosLast*/); //consider UTF encoding when cutting in the middle! (e.g. for macOS) @@ -263,18 +278,18 @@ void AFS::createFolderIfMissingRecursion(const AbstractPath& ap) //throw FileErr return; //already existing => possible, if createFolderIfMissingRecursion() is run in parallel } catch (FileError&) {} //not yet existing or access error - //catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } -> details needed??? throw; } } -std::optional<AFS::ItemType> AFS::itemStillExistsViaFolderTraversal(const AfsPath& afsPath) const //throw FileError +//default implementation: folder traversal +std::optional<AFS::ItemType> AFS::itemStillExists(const AfsPath& afsPath) const //throw FileError { try { - //fast check: 1. perf 2. expected by perfgetFolderStatusNonBlocking() + //fast check: 1. perf 2. expected by getFolderStatusNonBlocking() 3. traversing non-existing folder below MIGHT NOT FAIL (e.g. for SFTP on AWS) return getItemType(afsPath); //throw FileError } catch (const FileError& e) //not existing or access error @@ -289,13 +304,13 @@ std::optional<AFS::ItemType> AFS::itemStillExistsViaFolderTraversal(const AfsPat const Zstring itemName = getItemName(afsPath); assert(!itemName.empty()); - const std::optional<ItemType> parentType = itemStillExistsViaFolderTraversal(*parentAfsPath); //throw FileError + const std::optional<ItemType> parentType = AFS::itemStillExists(*parentAfsPath); //throw FileError if (parentType && *parentType != ItemType::FILE /*obscure, but possible (and not an error)*/) try { traverseFolderFlat(*parentAfsPath, //throw FileError - [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, - [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, + [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, + [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, [&](const SymlinkInfo& si) { if (si.itemName == itemName) throw ItemType::SYMLINK; }); } catch (const ItemType&) //finding the item after getItemType() previously failed is exceptional @@ -307,109 +322,106 @@ std::optional<AFS::ItemType> AFS::itemStillExistsViaFolderTraversal(const AfsPat } -void AFS::removeFolderIfExistsRecursion(const AbstractPath& ap, //throw FileError - const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion, //optional - const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) //one call for each object! +//default implementation: folder traversal +void AFS::removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const //one call for each object! { - warn_static("Support Google Drive simple recursive deletion") - - std::function<void(const AbstractPath& folderPath)> removeFolderRecursionImpl; - removeFolderRecursionImpl = [&onBeforeFileDeletion, &onBeforeFolderDeletion, &removeFolderRecursionImpl](const AbstractPath& folderPath) //throw FileError + //deferred recursion => save stack space and allow deletion of extremely deep hierarchies! + std::function<void(const AfsPath& folderPath)> removeFolderRecursionImpl; + removeFolderRecursionImpl = [this, &onBeforeFileDeletion, &onBeforeFolderDeletion, &removeFolderRecursionImpl](const AfsPath& folderPath) //throw FileError { - //deferred recursion => save stack space and allow deletion of extremely deep hierarchies! std::vector<Zstring> fileNames; std::vector<Zstring> folderNames; std::vector<Zstring> symlinkNames; - AFS::traverseFolderFlat(folderPath, //throw FileError - [&](const AFS::FileInfo& fi) { fileNames .push_back(fi.itemName); }, - [&](const AFS::FolderInfo& fi) { folderNames .push_back(fi.itemName); }, - [&](const AFS::SymlinkInfo& si) { symlinkNames.push_back(si.itemName); }); + traverseFolderFlat(folderPath, //throw FileError + [&](const FileInfo& fi) { fileNames.push_back(fi.itemName); }, + [&](const FolderInfo& fi) { folderNames.push_back(fi.itemName); }, + [&](const SymlinkInfo& si) { symlinkNames.push_back(si.itemName); }); for (const Zstring& fileName : fileNames) { - const AbstractPath filePath = AFS::appendRelPath(folderPath, fileName); + const AfsPath filePath(nativeAppendPaths(folderPath.value, fileName)); if (onBeforeFileDeletion) - onBeforeFileDeletion(AFS::getDisplayPath(filePath)); + onBeforeFileDeletion(getDisplayPath(filePath)); //throw X - AFS::removeFilePlain(filePath); //throw FileError + removeFilePlain(filePath); //throw FileError } for (const Zstring& symlinkName : symlinkNames) { - const AbstractPath linkPath = AFS::appendRelPath(folderPath, symlinkName); + const AfsPath linkPath(nativeAppendPaths(folderPath.value, symlinkName)); if (onBeforeFileDeletion) - onBeforeFileDeletion(AFS::getDisplayPath(linkPath)); //throw X + onBeforeFileDeletion(getDisplayPath(linkPath)); //throw X - AFS::removeSymlinkPlain(linkPath); //throw FileError + removeSymlinkPlain(linkPath); //throw FileError } for (const Zstring& folderName : folderNames) - removeFolderRecursionImpl(AFS::appendRelPath(folderPath, folderName)); //throw FileError + removeFolderRecursionImpl(AfsPath(nativeAppendPaths(folderPath.value, folderName))); //throw FileError if (onBeforeFolderDeletion) - onBeforeFolderDeletion(AFS::getDisplayPath(folderPath)); //throw X + onBeforeFolderDeletion(getDisplayPath(folderPath)); //throw X - AFS::removeFolderPlain(folderPath); //throw FileError + removeFolderPlain(folderPath); //throw FileError }; //-------------------------------------------------------------------------------------------------------------- warn_static("what about parallelOps?") //no error situation if directory is not existing! manual deletion relies on it! - if (std::optional<ItemType> type = AFS::itemStillExists(ap)) //throw FileError + if (std::optional<ItemType> type = itemStillExists(afsPath)) //throw FileError { if (*type == AFS::ItemType::SYMLINK) { if (onBeforeFileDeletion) - onBeforeFileDeletion(AFS::getDisplayPath(ap)); + onBeforeFileDeletion(getDisplayPath(afsPath)); //throw X - AFS::removeSymlinkPlain(ap); //throw FileError + removeSymlinkPlain(afsPath); //throw FileError } else - removeFolderRecursionImpl(ap); //throw FileError + removeFolderRecursionImpl(afsPath); //throw FileError } else //even if the folder did not exist anymore, significant I/O work was done => report - if (onBeforeFolderDeletion) onBeforeFolderDeletion(AFS::getDisplayPath(ap)); + if (onBeforeFolderDeletion) onBeforeFolderDeletion(getDisplayPath(afsPath)); //throw X } -bool AFS::removeFileIfExists(const AbstractPath& ap) //throw FileError +void AFS::removeFileIfExists(const AbstractPath& ap) //throw FileError { try { AFS::removeFilePlain(ap); //throw FileError - return true; } - catch (const FileError& e) + catch (const FileError&) { try { if (!AFS::itemStillExists(ap)) //throw FileError - return false; + return; } - catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } //unclear which exception is more relevant - + catch (const FileError& e2) { throw FileError(replaceCpy(_("Cannot delete file %x."), L"%x", fmtPath(getDisplayPath(ap))), replaceCpy(e2.toString(), L"\n\n", L"\n")); } + //more relevant than previous exception (which could be "item not found") throw; } } -bool AFS::removeSymlinkIfExists(const AbstractPath& ap) //throw FileError +void AFS::removeSymlinkIfExists(const AbstractPath& ap) //throw FileError { try { AFS::removeSymlinkPlain(ap); //throw FileError - return true; } - catch (const FileError& e) + catch (const FileError&) { try { if (!AFS::itemStillExists(ap)) //throw FileError - return false; + return; } - catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } //unclear which exception is more relevant - + catch (const FileError& e2) { throw FileError(replaceCpy(_("Cannot delete symbolic link %x."), L"%x", fmtPath(getDisplayPath(ap))), replaceCpy(e2.toString(), L"\n\n", L"\n")); } + //more relevant than previous exception (which could be "item not found") throw; } } @@ -421,14 +433,15 @@ void AFS::removeEmptyFolderIfExists(const AbstractPath& ap) //throw FileError { AFS::removeFolderPlain(ap); //throw FileError } - catch (const FileError& e) + catch (const FileError&) { try { if (!AFS::itemStillExists(ap)) //throw FileError return; } - catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } //unclear which exception is more relevant + catch (const FileError& e2) { throw FileError(replaceCpy(_("Cannot delete directory %x."), L"%x", fmtPath(getDisplayPath(ap))), replaceCpy(e2.toString(), L"\n\n", L"\n")); } + //more relevant than previous exception (which could be "item not found") throw; } diff --git a/FreeFileSync/Source/fs/abstract.h b/FreeFileSync/Source/fs/abstract.h index 682ddf9b..5821870c 100755..100644 --- a/FreeFileSync/Source/fs/abstract.h +++ b/FreeFileSync/Source/fs/abstract.h @@ -69,8 +69,8 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t static std::optional<Zstring> getNativeItemPath(const AbstractPath& ap) { return ap.afsDevice.ref().getNativeItemPath(ap.afsPath); } //---------------------------------------------------------------------------------------------------------------- - static void connectNetworkFolder(const AbstractPath& ap, bool allowUserInteraction) //throw FileError - { return ap.afsDevice.ref().connectNetworkFolder(ap.afsPath, allowUserInteraction); } + static void authenticateAccess(const AfsDevice& afsDevice, bool allowUserInteraction) //throw FileError + { return afsDevice.ref().authenticateAccess(allowUserInteraction); } static int getAccessTimeout(const AbstractPath& ap) { return ap.afsDevice.ref().getAccessTimeout(); } //returns "0" if no timeout in force @@ -81,35 +81,39 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t using FileId = zen::Zbase<char>; //AfsDevice-dependent unique ID - enum class ItemType + enum class ItemType : unsigned char { FILE, FOLDER, SYMLINK, }; //(hopefully) fast: does not distinguish between error/not existing + //root path? => do access test static ItemType getItemType(const AbstractPath& ap) { return ap.afsDevice.ref().getItemType(ap.afsPath); } //throw FileError //assumes: - base path still exists // - all child item path parts must correspond to folder traversal // => we can conclude whether an item is *not* existing anymore by doing a *case-sensitive* name search => potentially SLOW! + // root path? => do access test static std::optional<ItemType> itemStillExists(const AbstractPath& ap) { return ap.afsDevice.ref().itemStillExists(ap.afsPath); } //throw FileError //---------------------------------------------------------------------------------------------------------------- - //target existing: fail/ignore + //already existing: fail/ignore //does NOT create parent directories recursively if not existing static void createFolderPlain(const AbstractPath& ap) { ap.afsDevice.ref().createFolderPlain(ap.afsPath); } //throw FileError - //no error if already existing + //already existing: ignore //creates parent directories recursively if not existing static void createFolderIfMissingRecursion(const AbstractPath& ap); //throw FileError - static bool removeFileIfExists (const AbstractPath& ap); //throw FileError; return "false" if file is not existing - static bool removeSymlinkIfExists(const AbstractPath& ap); // - static void removeEmptyFolderIfExists(const AbstractPath& ap); //throw FileError static void removeFolderIfExistsRecursion(const AbstractPath& ap, //throw FileError - const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion, //optional - const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion); //one call for each object! + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) //one call for each object! + { return ap.afsDevice.ref().removeFolderIfExistsRecursion(ap.afsPath, onBeforeFileDeletion, onBeforeFolderDeletion); } + + static void removeFileIfExists (const AbstractPath& ap); //throw FileError; return "false" if file is not existing + static void removeSymlinkIfExists(const AbstractPath& ap); // + static void removeEmptyFolderIfExists(const AbstractPath& ap); //throw FileError static void removeFilePlain (const AbstractPath& ap) { ap.afsDevice.ref().removeFilePlain (ap.afsPath); } //throw FileError static void removeSymlinkPlain(const AbstractPath& ap) { ap.afsDevice.ref().removeSymlinkPlain(ap.afsPath); } //throw FileError @@ -125,7 +129,6 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t static zen::ImageHolder getThumbnailImage(const AbstractPath& ap, int pixelSize) { return ap.afsDevice.ref().getThumbnailImage(ap.afsPath, pixelSize); } //---------------------------------------------------------------------------------------------------------------- - struct StreamAttributes { time_t modTime; //number of seconds since Jan. 1st 1970 UTC @@ -235,10 +238,7 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t using TraverserWorkload = std::vector<std::pair<AfsPath, std::shared_ptr<TraverserCallback> /*throw X*/>>; //- client needs to handle duplicate file reports! (FilePlusTraverser fallback, retrying to read directory contents, ...) - static void traverseFolderRecursive(const AfsDevice& afsDevice, const TraverserWorkload& workload /*throw X*/, size_t parallelOps) - { - afsDevice.ref().traverseFolderRecursive(workload, parallelOps); //throw - } + static void traverseFolderRecursive(const AfsDevice& afsDevice, const TraverserWorkload& workload /*throw X*/, size_t parallelOps) { afsDevice.ref().traverseFolderRecursive(workload, parallelOps); } static void traverseFolderFlat(const AbstractPath& ap, //throw FileError const std::function<void (const FileInfo& fi)>& onFile, // @@ -276,7 +276,7 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t //accummulated delta != file size! consider ADS, sparse, compressed files const zen::IOCallback& notifyUnbufferedIO /*throw X*/); - //target existing: fail/ignore + //already existing: fail/ignore //symlink handling: follow link! static void copyNewFolder(const AbstractPath& apSource, const AbstractPath& apTarget, bool copyFilePermissions); //throw FileError @@ -309,7 +309,13 @@ struct AbstractFileSystem //THREAD-SAFETY: "const" member functions must model t protected: - std::optional<ItemType> itemStillExistsViaFolderTraversal(const AfsPath& afsPath) const; //throw FileError + //default implementation: folder traversal + virtual std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const = 0; //throw FileError + + //default implementation: folder traversal + virtual void removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const = 0; //one call for each object! void traverseFolderFlat(const AfsPath& afsPath, //throw FileError const std::function<void (const FileInfo& fi)>& onFile, // @@ -333,16 +339,16 @@ private: //---------------------------------------------------------------------------------------------------------------- virtual ItemType getItemType(const AfsPath& afsPath) const = 0; //throw FileError - virtual std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const = 0; //throw FileError //---------------------------------------------------------------------------------------------------------------- - //target existing: fail/ignore + //already existing: fail/ignore virtual void createFolderPlain(const AfsPath& afsPath) const = 0; //throw FileError //non-recursive folder deletion: virtual void removeFilePlain (const AfsPath& afsPath) const = 0; //throw FileError virtual void removeSymlinkPlain(const AfsPath& afsPath) const = 0; //throw FileError virtual void removeFolderPlain (const AfsPath& afsPath) const = 0; //throw FileError + //---------------------------------------------------------------------------------------------------------------- //virtual void setModTime(const AfsPath& afsPath, time_t modTime) const = 0; //throw FileError, follows symlinks @@ -382,7 +388,7 @@ private: virtual zen::ImageHolder getFileIcon (const AfsPath& afsPath, int pixelSize) const = 0; //noexcept; optional return value virtual zen::ImageHolder getThumbnailImage(const AfsPath& afsPath, int pixelSize) const = 0; // - virtual void connectNetworkFolder(const AfsPath& afsPath, bool allowUserInteraction) const = 0; //throw FileError + virtual void authenticateAccess(bool allowUserInteraction) const = 0; //throw FileError virtual int getAccessTimeout() const = 0; //returns "0" if no timeout in force @@ -435,9 +441,9 @@ AbstractFileSystem::OutputStream::~OutputStream() //we delete the file on errors: => file should not have existed prior to creating OutputStream instance!! outStream_.reset(); //close file handle *before* remove! - - if (!finalizeSucceeded_) //transactional output stream! => clean up! - //even needed for Google Drive: e.g. user might cancel during OutputStreamImpl::finalize(), just after file was written transactionally + + if (!finalizeSucceeded_) //transactional output stream! => clean up! + //even needed for Google Drive: e.g. user might cancel during OutputStreamImpl::finalize(), just after file was written transactionally try { AbstractFileSystem::removeFilePlain(filePath_); /*throw FileError*/ } catch (FileError& e) { (void)e; } } @@ -509,7 +515,7 @@ void AbstractFileSystem::copyNewFolder(const AbstractPath& apSource, const Abstr throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(getDisplayPath(apTarget))), _("Operation not supported for different base folder types.")); - //target existing: fail/ignore + //already existing: fail/ignore createFolderPlain(apTarget); //throw FileError } diff --git a/FreeFileSync/Source/fs/abstract_impl.h b/FreeFileSync/Source/fs/abstract_impl.h index 8f4a8d3d..585c074e 100755..100644 --- a/FreeFileSync/Source/fs/abstract_impl.h +++ b/FreeFileSync/Source/fs/abstract_impl.h @@ -46,6 +46,26 @@ std::wstring tryReportingDirError(Function cmd /*throw FileError*/, AbstractFile } } +template <class Command> inline +bool tryReportingItemError(Command cmd, AbstractFileSystem::TraverserCallback& callback, const Zstring& itemName) //throw X, return "true" on success, "false" if error was ignored +{ + for (size_t retryNumber = 0;; ++retryNumber) + try + { + cmd(); //throw FileError + return true; + } + catch (const zen::FileError& e) + { + switch (callback.reportItemError(e.toString(), retryNumber, itemName)) //throw X + { + case AbstractFileSystem::TraverserCallback::ON_ERROR_RETRY: + break; + case AbstractFileSystem::TraverserCallback::ON_ERROR_CONTINUE: + return false; + } + } +} //========================================================================================== /* @@ -102,6 +122,9 @@ public: //return "bytesToRead" bytes unless end of stream! size_t read(void* buffer, size_t bytesToRead) //throw <write error> { + if (bytesToRead == 0) //"read() with a count of 0 returns zero" => indistinguishable from end of file! => check! + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + zen::numberTo<std::string>(__LINE__)); + auto it = static_cast<std::byte*>(buffer); const auto itEnd = it + bytesToRead; @@ -189,178 +212,63 @@ private: //========================================================================================== -template <class Context, class Function> -struct Task -{ - Function getResult; //throw FileError - /* [[no_unique_address]] */ Context ctx; -}; - - -template <class Context, class Function> -struct TaskResult -{ - Task<Context, Function> wi; - std::exception_ptr error; //mutually exclusive - decltype(wi.getResult()) value; // -}; - -enum class SchedulerStatus -{ - HAVE_RESULT, - FINISHED, -}; - -template <class Context, class... Functions> //avoid std::function memory alloc + virtual calls -class TaskScheduler +//Google Drive/MTP happily create duplicate files/folders with the same names, without failing +//=> however, FFS's "check if already exists after failure" idiom *requires* failure +//=> serialize access (at path level) so that GoogleFileState access and file/folder creation act as a single operation +template <class NativePath> +class PathAccessLocker { public: - TaskScheduler(size_t threadCount, const std::string& groupName) : - threadGroup_(zen::ThreadGroup<std::function<void()>>(threadCount, groupName)) {} - - ~TaskScheduler() { threadGroup_ = {}; } //TaskScheduler must out-live threadGroup! (captured "this") - - //context of controlling thread, non-blocking: - template <class Function> - void run(Task<Context, Function>&& wi, bool insertFront = false) - { - threadGroup_->run([this, wi = std::move(wi)] - { - try { this->returnResult<Function>({ wi, nullptr, wi.getResult() }); } //throw FileError - catch (...) { this->returnResult<Function>({ wi, std::current_exception(), {} }); } - }, insertFront); - - std::lock_guard dummy(lockResult_); - ++resultsPending_; - } + PathAccessLocker() {} - //context of controlling thread, blocking: - SchedulerStatus getResults(std::tuple<std::vector<TaskResult<Context, Functions>>...>& results) + class Lock { - std::apply([](auto&... r) { (..., r.clear()); }, results); - - std::unique_lock dummy(lockResult_); - - auto resultsReady = [&] + public: + Lock(const NativePath& nativePath) //throw SysError { - bool ready = false; - std::apply([&ready](const auto&... r) { ready = (... || !r.empty()); }, results_); - return ready; - }; - - if (!resultsReady() && resultsPending_ == 0) - return SchedulerStatus::FINISHED; - - conditionNewResult_.wait(dummy, [&resultsReady] { return resultsReady(); }); - - results.swap(results_); //reuse memory + avoid needless item-level mutex locking - return SchedulerStatus::HAVE_RESULT; - } - -private: - TaskScheduler (const TaskScheduler&) = delete; - TaskScheduler& operator=(const TaskScheduler&) = delete; - - //context of worker threads, non-blocking: - template <class Function> - void returnResult(TaskResult<Context, Function>&& r) - { - { - std::lock_guard dummy(lockResult_); - - std::get<std::vector<TaskResult<Context, Function>>>(results_).push_back(std::move(r)); - --resultsPending_; + { + const std::shared_ptr<PathAccessLocker> gpalh = getGlobalInstance(); //throw SysError + if (!gpalh) + throw zen::SysError(L"Function call not allowed during process init/shutdown."); + m_ = gpalh->getOrCreateMutex(nativePath); + } + m_->lock(); } - conditionNewResult_.notify_all(); - } - - std::optional<zen::ThreadGroup<std::function<void()>>> threadGroup_; - - std::mutex lockResult_; - size_t resultsPending_ = 0; - std::tuple<std::vector<TaskResult<Context, Functions>>...> results_; - std::condition_variable conditionNewResult_; -}; - - -struct TravContext -{ - Zstring errorItemName; //empty if all items affected - size_t errorRetryCount = 0; - std::shared_ptr<AbstractFileSystem::TraverserCallback> cb; //call by controlling thread only! => don't require traverseFolderParallel() callbacks to be thread-safe! -}; - - -template <class... Functions> -class GenericDirTraverser -{ -public: - using Function1 = zen::GetFirstOfT<Functions...>; + ~Lock() { m_->unlock(); } - GenericDirTraverser(std::vector<Task<TravContext, Function1>>&& initialTasks /*throw X*/, size_t parallelOps, const std::string& threadGroupName) : - scheduler_(parallelOps, threadGroupName) - { - //set the initial work load - for (auto& item : initialTasks) - scheduler_.template run<Function1>(std::move(item)); + private: + Lock (const Lock&) = delete; + Lock& operator=(const Lock&) = delete; - //run loop - std::tuple<std::vector<TaskResult<TravContext, Functions>>...> results; //avoid per-getNextResults() memory allocations (=> swap instead!) - - while (scheduler_.getResults(results) == SchedulerStatus::HAVE_RESULT) - std::apply([&](auto&... r) { (..., this->evalResultList(r)); }, results); //throw X - } + std::shared_ptr<std::mutex> m_; + }; private: - GenericDirTraverser (const GenericDirTraverser&) = delete; - GenericDirTraverser& operator=(const GenericDirTraverser&) = delete; + PathAccessLocker (const PathAccessLocker&) = delete; + PathAccessLocker& operator=(const PathAccessLocker&) = delete; - template <class Function> - void evalResultList(std::vector<TaskResult<TravContext, Function>>& results /*throw X*/) + static std::shared_ptr<PathAccessLocker> getGlobalInstance(); + + std::shared_ptr<std::mutex> getOrCreateMutex(const NativePath& nativePath) { - for (TaskResult<TravContext, Function>& result : results) - evalResult(result); //throw X + std::shared_ptr<std::mutex> m; + pathLocks_.access([&](std::map<NativePath, std::weak_ptr<std::mutex>>& pathLocks) + { + //remove obsolete entries + zen::eraseIf(pathLocks, [](const auto& v) { return !v.second.lock(); }); + + //get or create mutex + std::weak_ptr<std::mutex>& weakPtr = pathLocks[nativePath]; + m = weakPtr.lock(); + if (!m) + weakPtr = m = std::make_shared<std::mutex>(); + }); + return m; } - template <class Function> - void evalResult(TaskResult<TravContext, Function>& result /*throw X*/); - - //specialize! - template <class Function> - void evalResultValue(const typename Function::Result& r, std::shared_ptr<AbstractFileSystem::TraverserCallback>& cb /*throw X*/); - - TaskScheduler<TravContext, Functions...> scheduler_; + zen::Protected<std::map<NativePath, std::weak_ptr<std::mutex>>> pathLocks_; }; - -template <class... Functions> -template <class Function> -void GenericDirTraverser<Functions...>::evalResult(TaskResult<TravContext, Function>& result /*throw X*/) -{ - auto& cb = result.wi.ctx.cb; - try - { - if (result.error) - std::rethrow_exception(result.error); //throw FileError - } - catch (const zen::FileError& e) - { - switch (result.wi.ctx.errorItemName.empty() ? - cb->reportDirError (e.toString(), result.wi.ctx.errorRetryCount) : //throw X - cb->reportItemError(e.toString(), result.wi.ctx.errorRetryCount, result.wi.ctx.errorItemName)) //throw X - { - case AbstractFileSystem::TraverserCallback::ON_ERROR_RETRY: - //user expects that the task is retried immediately => we can't do much about other errors already waiting in the queue, but at least *prepend* to the work load! - scheduler_.template run<Function>({ std::move(result.wi.getResult), TravContext{ result.wi.ctx.errorItemName, result.wi.ctx.errorRetryCount + 1, cb }}, - true /*insertFront*/); - return; - - case AbstractFileSystem::TraverserCallback::ON_ERROR_CONTINUE: - return; - } - } - evalResultValue<Function>(result.value, cb); //throw X -} } #endif //IMPL_HELPER_H_873450978453042524534234 diff --git a/FreeFileSync/Source/fs/concrete.cpp b/FreeFileSync/Source/fs/concrete.cpp index 337301db..8053e622 100755..100644 --- a/FreeFileSync/Source/fs/concrete.cpp +++ b/FreeFileSync/Source/fs/concrete.cpp @@ -6,17 +6,23 @@ #include "concrete.h" #include "native.h" + #include "ftp.h" + #include "sftp.h" + #include "gdrive.h" using namespace fff; void fff::initAfs(const AfsConfig& cfg) { + googleDriveInit(appendSeparator(cfg.configDirPathPf) + Zstr("GoogleDrive"), + appendSeparator(cfg.resourceDirPathPf) + Zstr("cacert.pem")); } void fff::teardownAfs() { + googleDriveTeardown(); } @@ -27,6 +33,14 @@ AbstractPath fff::createAbstractPath(const Zstring& itemPathPhrase) //noexcept return createItemPathNative(itemPathPhrase); //noexcept //then the rest: + if (acceptsItemPathPhraseFtp(itemPathPhrase)) //noexcept + return createItemPathFtp(itemPathPhrase); //noexcept + + if (acceptsItemPathPhraseSftp(itemPathPhrase)) //noexcept + return createItemPathSftp(itemPathPhrase); //noexcept + + if (acceptsItemPathPhraseGdrive(itemPathPhrase)) //noexcept + return createItemPathGdrive(itemPathPhrase); //noexcept //no idea? => native! diff --git a/FreeFileSync/Source/fs/concrete.h b/FreeFileSync/Source/fs/concrete.h index dec45300..3ff507dd 100755..100644 --- a/FreeFileSync/Source/fs/concrete.h +++ b/FreeFileSync/Source/fs/concrete.h @@ -13,8 +13,8 @@ namespace fff { struct AfsConfig { - Zstring configDirPathPf; //directory to store AFS-specific files Zstring resourceDirPathPf; //directory to read AFS-specific files + Zstring configDirPathPf; //directory to store AFS-specific files }; void initAfs(const AfsConfig& cfg); void teardownAfs(); diff --git a/FreeFileSync/Source/fs/ftp.cpp b/FreeFileSync/Source/fs/ftp.cpp new file mode 100644 index 00000000..653994ed --- /dev/null +++ b/FreeFileSync/Source/fs/ftp.cpp @@ -0,0 +1,2229 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "ftp.h" +#include <zen/basic_math.h> +#include <zen/sys_error.h> +#include <zen/globals.h> +#include <zen/time.h> +#include "libcurl/curl_wrap.h" //DON'T include <curl/curl.h> directly! +#include "init_curl_libssh2.h" +#include "ftp_common.h" +#include "abstract_impl.h" +#include "../base/resolve_path.h" + #include <gtk/gtk.h> + +using namespace zen; +using namespace fff; +using AFS = AbstractFileSystem; + + +namespace +{ +Zstring concatenateFtpFolderPathPhrase(const FtpLoginInfo& login, const AfsPath& afsPath); //noexcept + +const std::chrono::seconds FTP_SESSION_MAX_IDLE_TIME (20); +const std::chrono::seconds FTP_SESSION_CLEANUP_INTERVAL(4); +const int FTP_STREAM_BUFFER_SIZE = 512 * 1024; //unit: [byte] +//FTP stream buffer should be at least as big as the biggest AFS block size (currently 256 KB for MTP), +//but there seems to be no reason for an upper limit + +const Zchar ftpPrefix[] = Zstr("ftp:"); + +enum class ServerEncoding +{ + utf8, + ansi +}; + +//use all configuration data that *defines* an SFTP session as key when buffering sessions! This is what user expects, e.g. when changing settings in FTP login dialog +struct FtpSessionId +{ + /*explicit*/ FtpSessionId(const FtpLoginInfo& login) : + server(login.server), + port(login.port), + username(login.username), + password(login.password), + useSsl(login.useSsl) {} + + Zstring server; + int port = 0; + Zstring username; + Zstring password; + bool useSsl = false; + //timeoutSec => irrelevant for session equality +}; + + +bool operator<(const FtpSessionId& lhs, const FtpSessionId& rhs) +{ + //exactly the type of case insensitive comparison we need for server names! + int rv = compareAsciiNoCase(lhs.server, rhs.server); //https://msdn.microsoft.com/en-us/library/windows/desktop/ms738519#IDNs + if (rv != 0) + return rv < 0; + + if (lhs.port != rhs.port) + return lhs.port < rhs.port; + + rv = compareString(lhs.username, rhs.username); //case sensitive! + if (rv != 0) + return rv < 0; + + rv = compareString(lhs.password, rhs.password); //case sensitive! + if (rv != 0) + return rv < 0; + + return lhs.useSsl < rhs.useSsl; +} + + +Zstring ansiToUtfEncoding(const std::string& str) //throw SysError +{ + gsize bytesWritten = 0; //not including the terminating null + + GError* error = nullptr; + ZEN_ON_SCOPE_EXIT(if (error) ::g_error_free(error);); + + //https://developer.gnome.org/glib/stable/glib-Character-Set-Conversion.html#g-convert + gchar* utfStr = ::g_convert(str.c_str(), //const gchar* str, + str.size(), //gssize len, + "UTF-8", //const gchar* to_codeset, + "LATIN1", //const gchar* from_codeset, + nullptr, //gsize* bytes_read, + &bytesWritten, //gsize* bytes_written, + &error); //GError** error + if (!utfStr) + { + if (!error) + throw SysError(L"g_convert: unknown error. (" + utfTo<std::wstring>(str) + L")"); //user should never see this + + throw SysError(formatSystemError(L"g_convert", replaceCpy(_("Error Code %x"), L"%x", numberTo<std::wstring>(error->code)), + utfTo<std::wstring>(error->message)) + L" (" + utfTo<std::wstring>(str) + L")"); + } + ZEN_ON_SCOPE_EXIT(::g_free(utfStr)); + + return { utfStr, bytesWritten }; + + +} + + +std::string utfToAnsiEncoding(const Zstring& str) //throw SysError +{ + gsize bytesWritten = 0; //not including the terminating null + + GError* error = nullptr; + ZEN_ON_SCOPE_EXIT(if (error) ::g_error_free(error);); + + gchar* ansiStr = ::g_convert(str.c_str(), //const gchar* str, + str.size(), //gssize len, + "LATIN1", //const gchar* to_codeset, + "UTF-8", //const gchar* from_codeset, + nullptr, //gsize* bytes_read, + &bytesWritten, //gsize* bytes_written, + &error); //GError** error + if (!ansiStr) + { + if (!error) + throw SysError(L"g_convert: unknown error. (" + utfTo<std::wstring>(str) + L")"); //user should never see this + + throw SysError(formatSystemError(L"g_convert", replaceCpy(_("Error Code %x"), L"%x", numberTo<std::wstring>(error->code)), + utfTo<std::wstring>(error->message)) + L" (" + utfTo<std::wstring>(str) + L")"); + } + ZEN_ON_SCOPE_EXIT(::g_free(ansiStr)); + + return { ansiStr, bytesWritten }; + +} + + +Zstring serverToUtfEncoding(const std::string& str, ServerEncoding enc) //throw SysError +{ + switch (enc) + { + case ServerEncoding::utf8: + return utfTo<Zstring>(str); + case ServerEncoding::ansi: + return ansiToUtfEncoding(str); //throw SysError + } + assert(false); + return {}; +} + + +std::string utfToServerEncoding(const Zstring& str, ServerEncoding enc) //throw SysError +{ + switch (enc) + { + case ServerEncoding::utf8: + return utfTo<std::string>(str); + case ServerEncoding::ansi: + return utfToAnsiEncoding(str); //throw SysError + } + assert(false); + return {}; +} + + +std::wstring getCurlDisplayPath(const Zstring& serverName, const AfsPath& afsPath) +{ + Zstring displayPath = Zstring(ftpPrefix) + Zstr("//") + serverName; + const Zstring relPath = getServerRelPath(afsPath); + if (relPath != Zstr("/")) + displayPath += relPath; + return utfTo<std::wstring>(displayPath); +} + + +std::vector<std::string> splitFtpResponse(const std::string& buf) +{ + std::vector<std::string> lines; + + std::string lineBuf; + auto flushLineBuf = [&] + { + if (!lineBuf.empty()) + { + lines.push_back(lineBuf); + lineBuf.clear(); + } + }; + for (const char c : buf) + if (c == '\r' || c == '\n' || c == '\0') + flushLineBuf(); + else + lineBuf += c; + + flushLineBuf(); + return lines; +} + + +class FtpLineParser +{ +public: + FtpLineParser(const std::string& line) : line_(line), it_(line_.begin()) {} + + template <class Function> + std::string readRange(size_t count, Function acceptChar) //throw SysError + { + if (static_cast<ptrdiff_t>(count) > line_.end() - it_) + throw SysError(L"Unexpected end of line."); + + if (!std::all_of(it_, it_ + count, acceptChar)) + throw SysError(L"Expected char type not found."); + + std::string output(it_, it_ + count); + it_ += count; + return output; + } + + template <class Function> //expects non-empty range! + std::string readRange(Function acceptChar) //throw SysError + { + auto itEnd = std::find_if(it_, line_.end(), std::not_fn(acceptChar)); + std::string output(it_, itEnd); + if (output.empty()) + throw SysError(L"Expected char range not found."); + it_ = itEnd; + return output; + } + + char peekNextChar() const { return it_ == line_.end() ? '\0' : *it_; } + +private: + const std::string line_; + std::string::const_iterator it_; +}; + +//---------------------------------------------------------------------------------------------------------------- + +std::wstring tryFormatFtpErrorCode(int ec) //https://en.wikipedia.org/wiki/List_of_FTP_server_return_codes +{ + if (ec == 400) return L"The command was not accepted but the error condition is temporary."; + if (ec == 421) return L"Service not available, closing control connection."; + if (ec == 425) return L"Cannot open data connection."; + if (ec == 426) return L"Connection closed; transfer aborted."; + if (ec == 430) return L"Invalid username or password."; + if (ec == 431) return L"Need some unavailable resource to process security."; + if (ec == 434) return L"Requested host unavailable."; + if (ec == 450) return L"Requested file action not taken."; + if (ec == 451) return L"Local error in processing."; + if (ec == 452) return L"Insufficient storage space in system. File unavailable, e.g. file busy."; + if (ec == 500) return L"Syntax error, command unrecognized or command line too long."; + if (ec == 501) return L"Syntax error in parameters or arguments."; + if (ec == 502) return L"Command not implemented."; + if (ec == 503) return L"Bad sequence of commands."; + if (ec == 504) return L"Command not implemented for that parameter."; + if (ec == 521) return L"Data connection cannot be opened with this PROT setting."; + if (ec == 522) return L"Server does not support the requested network protocol."; + if (ec == 530) return L"User not logged in."; + if (ec == 532) return L"Need account for storing files."; + if (ec == 533) return L"Command protection level denied for policy reasons."; + if (ec == 534) return L"Could not connect to server; issue regarding SSL."; + if (ec == 535) return L"Failed security check."; + if (ec == 536) return L"Requested PROT level not supported by mechanism."; + if (ec == 537) return L"Command protection level not supported by security mechanism."; + if (ec == 550) return L"File unavailable, e.g. file not found, no access."; + if (ec == 551) return L"Requested action aborted. Page type unknown."; + if (ec == 552) return L"Requested file action aborted. Exceeded storage allocation."; + if (ec == 553) return L"File name not allowed."; + return L""; +} + +//================================================================================================================ +//================================================================================================================ + +Global<UniSessionCounter> globalFtpSessionCount(createUniSessionCounter()); + + +class FtpSession +{ +public: + FtpSession(const FtpSessionId& sessionId) : sessionId_(sessionId) //throw FileError + { + try + { + libsshCurlUnifiedInitCookie_ = getLibsshCurlUnifiedInitCookie(globalFtpSessionCount); //throw SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + } + + ~FtpSession() + { + if (easyHandle_) + ::curl_easy_cleanup(easyHandle_); + } + + //const FtpLoginInfo& getSessionId() const { return sessionId_; } + + struct Option + { + template <class T> + Option(CURLoption o, T val) : option(o), value(static_cast<uint64_t>(val)) { static_assert(sizeof(val) <= sizeof(value)); } + + template <class T> + Option(CURLoption o, T* val) : option(o), value(reinterpret_cast<uint64_t>(val)) { static_assert(sizeof(val) <= sizeof(value)); } + + CURLoption option = CURLOPT_LASTENTRY; + uint64_t value = 0; + }; + + //returns server response (header data) + std::string perform(const AfsPath* afsPath /*optional, use last-used path if null*/, bool isDir, + const std::vector<Option>& extraOptions, bool requiresUtf8, int timeoutSec) //throw FileError, SysError + { + if (requiresUtf8) //avoid endless recusion + sessionEnableUtf8(timeoutSec); //throw FileError + + if (!easyHandle_) + { + easyHandle_ = ::curl_easy_init(); + if (!easyHandle_) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"curl_easy_init", formatCurlErrorRaw(CURLE_OUT_OF_MEMORY), std::wstring())); + } + else + ::curl_easy_reset(easyHandle_); + + + std::vector<Option> options; + + curlErrorBuf_[0] = '\0'; + options.emplace_back(CURLOPT_ERRORBUFFER, curlErrorBuf_); + + headerData_.clear(); + using CbType = size_t (*)(const char* buffer, size_t size, size_t nitems, void* callbackData); + CbType onHeaderReceived = [](const char* buffer, size_t size, size_t nitems, void* callbackData) + { + auto& output = *static_cast<std::string*>(callbackData); + output.append(buffer, size * nitems); + return size * nitems; + }; + options.emplace_back(CURLOPT_HEADERDATA, &headerData_); + options.emplace_back(CURLOPT_HEADERFUNCTION, onHeaderReceived); + + std::string curlPath; //lifetime: keep alive until after curl_easy_setopt() below + if (std::any_of(extraOptions.begin(), extraOptions.end(), [](const Option& opt) { return opt.option == CURLOPT_FTP_FILEMETHOD && opt.value == CURLFTPMETHOD_NOCWD; })) + { + //CURLFTPMETHOD_NOCWD case => CURLOPT_URL will not be used for CWD but as argument, e.g., for MLSD + //curl was fixed to expect encoded paths in this case, too: https://github.com/curl/curl/issues/1974 + AfsPath targetPath; + bool targetPathisDir = true; + if (afsPath) + { + targetPath = *afsPath; + targetPathisDir = isDir; + } + curlPath = getCurlUrlPath(targetPath, targetPathisDir, timeoutSec); //throw FileError + workingDirPath_ = AfsPath(); + } + else + { + AfsPath currentPath; + bool currentPathisDir = true; + if (afsPath) + { + currentPath = *afsPath; + currentPathisDir = isDir; + } + else //try to use libcurl's last-used working dir and avoid excess CWD round trips + if (getActiveSocket()) //throw FileError + currentPath = workingDirPath_; + //what if our last curl_easy_perform() just deleted the working directory???? + //=> 1. libcurl recognizes last-used path and avoids the CWD accordingly 2. commands that depend on the working directory, e.g. PWD will fail on *some* servers + + curlPath = getCurlUrlPath(currentPath, currentPathisDir, timeoutSec); //throw FileError + workingDirPath_ = currentPathisDir ? currentPath : AfsPath(beforeLast(currentPath.value, FILE_NAME_SEPARATOR, IF_MISSING_RETURN_NONE)); + //remember libcurl's working dir: path might not exist => make sure to clear if ::curl_easy_perform() fails! + } + options.emplace_back(CURLOPT_URL, curlPath.c_str()); + + + const auto username = utfTo<std::string>(sessionId_.username); + const auto password = utfTo<std::string>(sessionId_.password); + if (!username.empty()) //else: libcurl handles anonymous login for us (including fake email as password) + { + options.emplace_back(CURLOPT_USERNAME, username.c_str()); + options.emplace_back(CURLOPT_PASSWORD, password.c_str()); + } + + if (sessionId_.port > 0) + options.emplace_back(CURLOPT_PORT, static_cast<long>(sessionId_.port)); + + options.emplace_back(CURLOPT_NOSIGNAL, 1L); //thread-safety: https://curl.haxx.se/libcurl/c/threadsafe.html + + options.emplace_back(CURLOPT_CONNECTTIMEOUT, timeoutSec); + + //CURLOPT_TIMEOUT: "Since this puts a hard limit for how long time a request is allowed to take, it has limited use in dynamic use cases with varying transfer times." + options.emplace_back(CURLOPT_LOW_SPEED_TIME, timeoutSec); + options.emplace_back(CURLOPT_LOW_SPEED_LIMIT, 1L); //[bytes], can't use "0" which means "inactive", so use some low number + + //"while libcurl is waiting for a [FTP] response, this value overrides CURLOPT_TIMEOUT." + options.emplace_back(CURLOPT_FTP_RESPONSE_TIMEOUT, timeoutSec); + + //CURLOPT_ACCEPTTIMEOUT_MS? => only relevant for "active" FTP connections + + if (!std::any_of(extraOptions.begin(), extraOptions.end(), [](const Option& opt) { return opt.option == CURLOPT_FTP_FILEMETHOD; })) + options.emplace_back(CURLOPT_FTP_FILEMETHOD, CURLFTPMETHOD_SINGLECWD); + //let's save these needless round trips!! most servers should support "CWD /folder/subfolder" + //=> 15% faster folder traversal time compared to CURLFTPMETHOD_MULTICWD! + //CURLFTPMETHOD_NOCWD? Already set in the MLSD case; but use for legacy servers, too? supported? + + + //Use share interface? https://curl.haxx.se/libcurl/c/libcurl-share.html + //perf test, 4 and 8 parallel threads: + // CURL_LOCK_DATA_DNS => no measurable total time difference + // CURL_LOCK_DATA_SSL_SESSION => freefilesync.org; not working at all: lots of CURLE_RECV_ERROR (seems nobody ever tested this with truly parallel FTP accesses!) +#if 0 + do not include this into release! + static CURLSH* curlShare = [] + { + struct ShareLocks + { + std::mutex lockIntenal; + std::mutex lockDns; + std::mutex lockSsl; + }; + static ShareLocks globalLocksTestingOnly; + + using LockFunType = void (*)(CURL* handle, curl_lock_data data, curl_lock_access access, void* userptr); //needed for cdecl function pointer cast + LockFunType lockFun = [](CURL* handle, curl_lock_data data, curl_lock_access access, void* userptr) + { + auto& locks = *static_cast<ShareLocks*>(userptr); + switch (data) + { + case CURL_LOCK_DATA_SHARE: + return locks.lockIntenal.lock(); + case CURL_LOCK_DATA_DNS: + return locks.lockDns.lock(); + case CURL_LOCK_DATA_SSL_SESSION: + return locks.lockSsl.lock(); + } + assert(false); + }; + using UnlockFunType = void (*)(CURL *handle, curl_lock_data data, void* userptr); + UnlockFunType unlockFun = [](CURL *handle, curl_lock_data data, void* userptr) + { + auto& locks = *static_cast<ShareLocks*>(userptr); + switch (data) + { + case CURL_LOCK_DATA_SHARE: + return locks.lockIntenal.unlock(); + case CURL_LOCK_DATA_DNS: + return locks.lockDns.unlock(); + case CURL_LOCK_DATA_SSL_SESSION: + return locks.lockSsl.unlock(); + } + assert(false); + }; + + CURLSH* cs = ::curl_share_init(); + assert(cs); + CURLSHcode rc = CURLSHE_OK; + rc = ::curl_share_setopt(cs, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS); + assert(rc == CURLSHE_OK); + rc = ::curl_share_setopt(cs, CURLSHOPT_SHARE, CURL_LOCK_DATA_SSL_SESSION); //buggy!? + assert(rc == CURLSHE_OK); + rc = ::curl_share_setopt(cs, CURLSHOPT_LOCKFUNC, lockFun); + assert(rc == CURLSHE_OK); + rc = ::curl_share_setopt(cs, CURLSHOPT_UNLOCKFUNC, unlockFun); + assert(rc == CURLSHE_OK); + rc = ::curl_share_setopt(cs, CURLSHOPT_USERDATA, &globalLocksTestingOnly); + assert(rc == CURLSHE_OK); + return cs; + }(); + //CURLSHcode ::curl_share_cleanup(curlShare); + options.emplace_back(CURLOPT_SHARE, curlShare); +#endif + + //TODO: FTP option to require certificate checking? +#if 0 + options.emplace_back(CURLOPT_CAINFO, "cacert.pem"); //hopefully latest version from https://curl.haxx.se/docs/caextract.html + //libcurl forwards this char-string to OpenSSL as is, which (thank god) accepts UTF8 +#else + options.emplace_back(CURLOPT_CAINFO, 0L); //be explicit: "even when [CURLOPT_SSL_VERIFYPEER] is disabled [...] curl may still load the certificate file specified in CURLOPT_CAINFO." + + //check if server certificate can be trusted? (Default: 1L) + // => may fail with: CURLE_PEER_FAILED_VERIFICATION: SSL certificate problem: certificate has expired + options.emplace_back(CURLOPT_SSL_VERIFYPEER, 0L); + //check that server name matches the name in the certificate? (Default: 2L) + // => may fail with: CURLE_PEER_FAILED_VERIFICATION: SSL: no alternative certificate subject name matches target host name 'freefilesync.org' + options.emplace_back(CURLOPT_SSL_VERIFYHOST, 0L); +#endif + if (sessionId_.useSsl) //https://tools.ietf.org/html/rfc4217 + { + options.emplace_back(CURLOPT_USE_SSL, CURLUSESSL_ALL); //require SSL for both control and data + options.emplace_back(CURLOPT_FTPSSLAUTH, CURLFTPAUTH_TLS); //try TLS first, then SSL (currently: CURLFTPAUTH_DEFAULT == CURLFTPAUTH_SSL) + } + + //let's not hold our breath until Curl adds a reasonable PASV handling => patch libcurl accordingly! + //https://github.com/curl/curl/issues/1455 + //https://github.com/curl/curl/pull/1470 + //support broken servers like this one: https://freefilesync.org/forum/viewtopic.php?t=4301 + + append(options, extraOptions); + + for (const Option& opt : options) + { + const CURLcode rc = ::curl_easy_setopt(easyHandle_, opt.option, opt.value); + if (rc != CURLE_OK) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"curl_easy_setopt " + numberTo<std::wstring>(opt.option), + formatCurlErrorRaw(rc), utfTo<std::wstring>(::curl_easy_strerror(rc)))); + } + + //======================================================================================================= + const CURLcode rcPerf = ::curl_easy_perform(easyHandle_); + //WTF: curl_easy_perform() considers FTP response codes 4XX, 5XX as failure, but for HTTP response codes 4XX are considered success!! CONSISTENCY, people!!! + long ftpStatus = 0; //optional + ::curl_easy_getinfo(easyHandle_, CURLINFO_RESPONSE_CODE, &ftpStatus); + //note: CURLOPT_FAILONERROR(default:off) is only available for HTTP + + //assert((rcPerf == CURLE_OK && 100 <= ftpStatus && ftpStatus < 400) || -> insufficient *FEAT can fail with 550, but still CURLE_OK because of * + // (rcPerf != CURLE_OK && (ftpStatus == 0 || 400 <= ftpStatus && ftpStatus < 600))); + //======================================================================================================= + + if (rcPerf != CURLE_OK) + { + workingDirPath_ = AfsPath(); //not sure what went wrong; no idea where libcurl's working dir currently is => libcurl might even have closed the old session! + throw SysError(formatLastCurlError(L"curl_easy_perform", rcPerf, ftpStatus)); + } + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + return headerData_; + } + + //returns server response (header data) + std::string runSingleFtpCommand(const std::string& ftpCmd, bool requiresUtf8, int timeoutSec) //throw FileError, SysError + { + struct curl_slist* quote = nullptr; + ZEN_ON_SCOPE_EXIT(::curl_slist_free_all(quote)); + quote = ::curl_slist_append(quote, ftpCmd.c_str()); + + std::vector<FtpSession::Option> options = + { + FtpSession::Option(CURLOPT_NOBODY, 1L), + FtpSession::Option(CURLOPT_QUOTE, quote), + }; + + //observation: libcurl sends CWD *after* CURLOPT_QUOTE has run + //perf: we neither need nor want libcurl to send CWD + return perform(nullptr /*re-use last-used path*/, true /*isDir*/, options, requiresUtf8, timeoutSec); //throw FileError, SysError + } + + //------------------------------------------------------------------------------------------------------------ + + bool supportsMlsd(int timeoutSec) { return getFeatureSupport(&Features::mlsd, timeoutSec); } // + bool supportsMlst(int timeoutSec) { return getFeatureSupport(&Features::mlst, timeoutSec); } // + bool supportsMfmt(int timeoutSec) { return getFeatureSupport(&Features::mfmt, timeoutSec); } //throw FileError + bool supportsClnt(int timeoutSec) { return getFeatureSupport(&Features::clnt, timeoutSec); } // + bool supportsUtf8(int timeoutSec) { return getFeatureSupport(&Features::utf8, timeoutSec); } // + + ServerEncoding getServerEncoding(int timeoutSec) { return supportsUtf8(timeoutSec) ? ServerEncoding::utf8 : ServerEncoding::ansi; } //throw FileError + + bool isHealthy() const + { + return numeric::dist(std::chrono::steady_clock::now(), lastSuccessfulUseTime_) <= FTP_SESSION_MAX_IDLE_TIME; + } + + std::string getServerRelPathInternal(const AfsPath& afsPath, int timeoutSec) //throw FileError + { + const Zstring serverRelPath = getServerRelPath(afsPath); + + if (afsPath.value.empty()) //endless recursion caveat!! getServerEncoding() transitively depends on getServerRelPathInternal() + return utfTo<std::string>(serverRelPath); + + const ServerEncoding encoding = getServerEncoding(timeoutSec); //throw FileError + try + { + return utfToServerEncoding(serverRelPath, encoding); //throw SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + } + +private: + FtpSession (const FtpSession&) = delete; + FtpSession& operator=(const FtpSession&) = delete; + + std::string getCurlUrlPath(const AfsPath& afsPath, bool isDir, int timeoutSec) //throw FileError + { + std::string serverRelPath; + + for (const std::string& comp : split(getServerRelPathInternal(afsPath, timeoutSec), '/', SplitType::ALLOW_EMPTY)) //throw FileError + { + char* compFmt = ::curl_easy_escape(easyHandle_, comp.c_str(), static_cast<int>(comp.size())); + if (!compFmt) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + replaceCpy<std::wstring>(L"curl_easy_escape: conversion failure (%x)", L"%x", utfTo<std::wstring>(comp))); + ZEN_ON_SCOPE_EXIT(::curl_free(compFmt)); + + serverRelPath += compFmt; + serverRelPath += '/'; + } + if (!serverRelPath.empty()) + serverRelPath.pop_back(); + + std::string path = utfTo<std::string>(Zstring(ftpPrefix) + Zstr("//") + sessionId_.server) + serverRelPath; + if (isDir && !endsWith(path, '/')) //curl-FTP needs directory paths to end with a slash + path += "/"; + return path; + } + + void sessionEnableUtf8(int timeoutSec) //throw FileError + { + //Some RFC-2640-non-compliant servers require UTF8 to be explicitly enabled: https://wiki.filezilla-project.org/Character_Encoding#Conflicting_specification + //e.g. this one (Microsoft FTP Service): https://freefilesync.org/forum/viewtopic.php?t=4303 + if (supportsUtf8(timeoutSec)) //throw FileError + { + //hopyfully libcurl will offer a better solution: https://github.com/curl/curl/issues/1457 + + //"OPTS UTF8 ON" needs to be activated each time libcurl internally creates a new session + + //[!] supportsUtf8() is buffered! => FTP session might not yet exist (or was closed by libcurl after a failure) + if (std::optional<curl_socket_t> currentSocket = getActiveSocket()) //throw FileError + if (*currentSocket == utf8EnabledSocket_) //caveat: a non-utf8-enabled session might already exist, e.g. from a previous call to supportsMlsd() + return; + try + { + //some servers even require "CLNT" before accepting "OPTS UTF8 ON": https://social.msdn.microsoft.com/Forums/en-US/d602574f-8a69-4d69-b337-52b6081902cf/problem-with-ftpwebrequestopts-utf8-on-501-please-clnt-first + if (supportsClnt(timeoutSec)) //throw FileError + runSingleFtpCommand("CLNT FreeFileSync", false /*requiresUtf8*/, timeoutSec); //throw FileError, SysError + + //"prefix the command with an asterisk to make libcurl continue even if the command fails" + //-> ignore if server does not know this legacy command (but report all *other* issues; else getActiveSocket() below won't return value and hide real error!) + runSingleFtpCommand("*OPTS UTF8 ON", false /*requiresUtf8*/, timeoutSec); //throw FileError, SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + + //make sure our unicode-enabled session is still there (== libcurl behaves as we expect) + std::optional<curl_socket_t> currentSocket = getActiveSocket(); //throw FileError + if (!currentSocket) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), L"Curl failed to cache FTP session."); //why is libcurl not caching the session??? + + utf8EnabledSocket_ = *currentSocket; //remember what we did + } + } + + std::optional<curl_socket_t> getActiveSocket() //throw FileError + { + if (easyHandle_) + { + curl_socket_t currentSocket = 0; + const CURLcode rc = ::curl_easy_getinfo(easyHandle_, CURLINFO_ACTIVESOCKET, ¤tSocket); + if (rc != CURLE_OK) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"curl_easy_getinfo: CURLINFO_ACTIVESOCKET", formatCurlErrorRaw(rc), utfTo<std::wstring>(::curl_easy_strerror(rc)))); + if (currentSocket != CURL_SOCKET_BAD) + return currentSocket; + } + return {}; + } + + struct Features + { + bool mlsd = false; + bool mlst = false; + bool mfmt = false; + bool clnt = false; + bool utf8 = false; + }; + using FeatureList = std::map<Zstring /*server name*/, std::optional<Features>, LessAsciiNoCase>; + + bool getFeatureSupport(bool Features::* status, int timeoutSec) //throw FileError + { + if (!featureCache_) + { + static FunStatGlobal<Protected<FeatureList>> globalServerFeatures; + globalServerFeatures.initOnce([] { return std::make_unique<Protected<FeatureList>>(); }); + + const auto sf = globalServerFeatures.get(); + if (!sf) + throw FileError(replaceCpy(_("Failed to get information about server %x."), L"%x", fmtPath(sessionId_.server)), + L"Function call not allowed during process shutdown."); + + sf->access([&](FeatureList& feat) { featureCache_ = feat[sessionId_.server]; }); + + if (!featureCache_) + { + std::string featResponse; + try + { + //ignore errors if server does not support FEAT (do those exist?), but fail for all others + featResponse = runSingleFtpCommand("*FEAT", false /*requiresUtf8*/, timeoutSec); //throw FileError, SysError + //used by sessionEnableUtf8()! => requiresUtf8 = false!!! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Failed to get information about server %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + sf->access([&](FeatureList& feat) + { + auto& f = feat[sessionId_.server]; + f = parseFeatResponse(featResponse); + featureCache_ = f; + }); + } + } + return (*featureCache_).*status; + } + + static Features parseFeatResponse(const std::string& featResponse) + { + Features output; //FEAT command: https://tools.ietf.org/html/rfc2389#page-4 + const std::vector<std::string> lines = splitFtpResponse(featResponse); + + auto it = std::find_if(lines.begin(), lines.end(), [](const std::string& line) { return startsWith(line, "211-"); }); + if (it != lines.end()) ++it; + for (; it != lines.end(); ++it) + { + const std::string& line = *it; + if (equalAsciiNoCase(line, "211 End")) + break; + + //https://tools.ietf.org/html/rfc3659#section-7.8 + //"a server-FTP process that supports MLST, and MLSD [...] MUST indicate that this support exists" + //"there is no distinct FEAT output for MLSD. The presence of the MLST feature indicates that both MLST and MLSD are supported" + if (equalAsciiNoCase (line, " MLST") || + startsWithAsciiNoCase(line, " MLST ")) //SP "MLST" [SP factlist] CRLF + output.mlsd = output.mlst = true; + + //https://tools.ietf.org/html/draft-somers-ftp-mfxx-04#section-3.3 + //"Where a server-FTP process supports the MFMT command [...] it MUST include the response to the FEAT command" + else if (equalAsciiNoCase(line, " MFMT")) //SP "MFMT" CRLF + output.mfmt = true; + + else if (equalAsciiNoCase(line, " UTF8")) + output.utf8 = true; + + else if (equalAsciiNoCase(line, " CLNT")) + output.clnt = true; + } + return output; + } + + std::wstring formatLastCurlError(const std::wstring& functionName, CURLcode ec, long ftpResponse) const + { + std::wstring errorMsg; + + if (curlErrorBuf_[0] != 0) + errorMsg = trimCpy(utfTo<std::wstring>(curlErrorBuf_)); + + if (ec != CURLE_RECV_ERROR) + { + const std::vector<std::string> headerLines = splitFtpResponse(headerData_); + if (!headerLines.empty()) + errorMsg += (errorMsg.empty() ? L"" : L"\n") + trimCpy(utfTo<std::wstring>(headerLines.back())); //that *should* be the servers error response + } + else //failed to get server response + { + const std::wstring descr = tryFormatFtpErrorCode(ftpResponse); + if (!descr.empty()) + errorMsg += (errorMsg.empty() ? L"" : L"\n") + numberTo<std::wstring>(ftpResponse) + L": " + descr; + } +#if 0 + //utfTo<std::wstring>(::curl_easy_strerror(ec)) is uninteresting + //use CURLINFO_OS_ERRNO ?? https://curl.haxx.se/libcurl/c/CURLINFO_OS_ERRNO.html + long nativeErrorCode = 0; + if (::curl_easy_getinfo(easyHandle_, CURLINFO_OS_ERRNO, &nativeErrorCode) == CURLE_OK) + if (nativeErrorCode != 0) + errorMsg += (errorMsg.empty() ? L"" : L"\n") + std::wstring(L"Native error code: ") + numberTo<std::wstring>(nativeErrorCode); +#endif + return formatSystemError(functionName, formatCurlErrorRaw(ec), errorMsg); + } + + const FtpSessionId sessionId_; + CURL* easyHandle_ = nullptr; + char curlErrorBuf_[CURL_ERROR_SIZE] = {}; + std::string headerData_; + + AfsPath workingDirPath_; + + curl_socket_t utf8EnabledSocket_ = 0; + + std::chrono::steady_clock::time_point lastSuccessfulUseTime_; + std::shared_ptr<UniCounterCookie> libsshCurlUnifiedInitCookie_; + + std::optional<Features> featureCache_; +}; + +//================================================================================================================ +//================================================================================================================ + +class FtpSessionManager //reuse (healthy) FTP sessions globally +{ + using IdleFtpSessions = std::vector<std::unique_ptr<FtpSession>>; + +public: + FtpSessionManager() : sessionCleaner_([this] + { + setCurrentThreadName("Session Cleaner[FTP]"); + runGlobalSessionCleanUp(); /*throw ThreadInterruption*/ + }) {} + ~FtpSessionManager() + { + sessionCleaner_.interrupt(); + sessionCleaner_.join(); + } + + void access(const FtpLoginInfo& login, const std::function<void(FtpSession& session)>& useFtpSession /*throw X*/) //throw FileError, X + { + Protected<IdleFtpSessions>& sessionStore = getSessionStore(login); + + std::unique_ptr<FtpSession> ftpSession; + + sessionStore.access([&](IdleFtpSessions& sessions) + { + //assume "isHealthy()" to avoid hitting server connection limits: (clean up of !isHealthy() after use, idle sessions via worker thread) + if (!sessions.empty()) + { + ftpSession = std::move(sessions.back ()); + /**/ sessions.pop_back(); + } + }); + + //create new FTP session outside the lock: 1. don't block other threads 2. non-atomic regarding "sessionStore"! => one session too many is not a problem! + if (!ftpSession) + ftpSession = std::make_unique<FtpSession>(login); //throw FileError + + ZEN_ON_SCOPE_EXIT( + if (ftpSession->isHealthy()) //thread that created the "!isHealthy()" session is responsible for clean up (avoid hitting server connection limits!) + sessionStore.access([&](IdleFtpSessions& sessions) { sessions.push_back(std::move(ftpSession)); }); ); + + useFtpSession(*ftpSession); //throw X + } + +private: + FtpSessionManager (const FtpSessionManager&) = delete; + FtpSessionManager& operator=(const FtpSessionManager&) = delete; + + Protected<IdleFtpSessions>& getSessionStore(const FtpSessionId& sessionId) + { + //single global session store per login; life-time bound to globalInstance => never remove a sessionStore!!! + Protected<IdleFtpSessions>* store = nullptr; + + globalSessionStore_.access([&](GlobalFtpSessions& sessionsById) + { + store = &sessionsById[sessionId]; //get or create + }); + static_assert(std::is_same_v<GlobalFtpSessions, std::map<FtpSessionId, Protected<IdleFtpSessions>>>, "require std::map so that the pointers we return remain stable"); + + return *store; + } + + //run a dedicated clean-up thread => it's unclear when the server let's a connection time out, so we do it preemptively + //context of worker thread: + void runGlobalSessionCleanUp() //throw ThreadInterruption + { + std::chrono::steady_clock::time_point lastCleanupTime; + for (;;) + { + const auto now = std::chrono::steady_clock::now(); + + if (now < lastCleanupTime + FTP_SESSION_CLEANUP_INTERVAL) + interruptibleSleep(lastCleanupTime + FTP_SESSION_CLEANUP_INTERVAL - now); //throw ThreadInterruption + + lastCleanupTime = std::chrono::steady_clock::now(); + + std::vector<Protected<IdleFtpSessions>*> sessionStores; //pointers remain stable, thanks to std::map<> + + globalSessionStore_.access([&](GlobalFtpSessions& sessionsById) + { + for (auto& [sessionId, idleSession] : sessionsById) + sessionStores.push_back(&idleSession); + }); + + for (Protected<IdleFtpSessions>* sessionStore : sessionStores) + for (bool done = false; !done;) + sessionStore->access([&](IdleFtpSessions& sessions) + { + for (std::unique_ptr<FtpSession>& sshSession : sessions) + if (!sshSession->isHealthy()) //!isHealthy() sessions are destroyed after use => in this context this means they have been idle for too long + { + sshSession.swap(sessions.back()); + /**/ sessions.pop_back(); //run ~FtpSession *inside* the lock! => avoid hitting server limits! + std::this_thread::yield(); + return; //don't hold lock for too long: delete only one session at a time, then yield... + } + done = true; + }); + } + } + + using GlobalFtpSessions = std::map<FtpSessionId, Protected<IdleFtpSessions>>; + + Protected<GlobalFtpSessions> globalSessionStore_; + InterruptibleThread sessionCleaner_; +}; + +//-------------------------------------------------------------------------------------- +UniInitializer globalStartupInitFtp(*globalFtpSessionCount.get()); //static ordering: place *before* SftpSessionManager instance! + +Global<FtpSessionManager> globalFtpSessionManager(std::make_unique<FtpSessionManager>()); +//-------------------------------------------------------------------------------------- + +void accessFtpSession(const FtpLoginInfo& login, const std::function<void(FtpSession& session)>& useFtpSession /*throw X*/) //throw FileError, X +{ + if (const std::shared_ptr<FtpSessionManager> mgr = globalFtpSessionManager.get()) + mgr->access(login, useFtpSession); //throw FileError, X + else + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(login.server)), L"Function call not allowed during process init/shutdown."); +} + +//=========================================================================================================================== + +struct FtpItem +{ + AFS::ItemType type = AFS::ItemType::FILE; + Zstring itemName; + uint64_t fileSize = 0; + time_t modTime = 0; +}; + + +class FtpDirectoryReader +{ +public: + static std::vector<FtpItem> execute(const FtpLoginInfo& login, const AfsPath& afsDirPath) //throw FileError + { + std::string rawListing; //get raw FTP directory listing + + using CbType = size_t (*)(const char* buffer, size_t size, size_t nitems, void* callbackData); + CbType onBytesReceived = [](const char* buffer, size_t size, size_t nitems, void* callbackData) + { + auto& listing = *static_cast<std::string*>(callbackData); + listing.append(buffer, size * nitems); + return size * nitems; + //folder reading might take up to a minute in extreme cases (50,000 files): https://freefilesync.org/forum/viewtopic.php?t=5312 + }; + + std::vector<FtpItem> output; + + accessFtpSession(login, [&](FtpSession& session) //throw FileError + { + std::vector<FtpSession::Option> options = + { + FtpSession::Option(CURLOPT_WRITEDATA, &rawListing), + FtpSession::Option(CURLOPT_WRITEFUNCTION, onBytesReceived), + }; + + if (session.supportsMlsd(login.timeoutSec)) //throw FileError + { + options.emplace_back(CURLOPT_CUSTOMREQUEST, "MLSD"); + + //some FTP servers abuse https://tools.ietf.org/html/rfc3659#section-7.1 + //and process wildcards characters inside the "dirpath"; see http://www.proftpd.org/docs/howto/Globbing.html + // [] matches any character in the character set enclosed in the brackets + // * (not between brackets) matches any string, including the empty string + // ? (not between brackets) matches any single character + // + //of course this "helpfulness" blows up with MLSD + paths that incidentally contain wildcards: https://freefilesync.org/forum/viewtopic.php?t=5575 + const bool pathHasWildcards = [&] //=> globbing is reproducible even with freefilesync.org's FTP! + { + const size_t pos = afsDirPath.value.find(Zstr('[')); + if (pos != Zstring::npos) + if (afsDirPath.value.find(Zstr(']'), pos + 1) != Zstring::npos) + return true; + + return contains(afsDirPath.value, Zstr('*')) || + /**/ contains(afsDirPath.value, Zstr('?')); + }(); + + if (!pathHasWildcards) + options.emplace_back(CURLOPT_FTP_FILEMETHOD, CURLFTPMETHOD_NOCWD); //16% faster traversal compared to CURLFTPMETHOD_SINGLECWD (35% faster than CURLFTPMETHOD_MULTICWD) + } + //else: use "LIST" + CURLFTPMETHOD_SINGLECWD + + try + { + session.perform(&afsDirPath, true /*isDir*/, options, true /*requiresUtf8*/, login.timeoutSec); //throw FileError, SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot open directory %x."), L"%x", fmtPath(getCurlDisplayPath(login.server, afsDirPath))), e.toString()); } + + const ServerEncoding encoding = session.getServerEncoding(login.timeoutSec); //throw FileError + try + { + if (session.supportsMlsd(login.timeoutSec)) //throw FileError + output = parseMlsd(rawListing, encoding); //throw SysError + else + output = parseUnknown(rawListing, encoding); //throw SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read directory %x."), L"%x", fmtPath(getCurlDisplayPath(login.server, afsDirPath))), e.toString()); } + }); + return output; + } + +private: + FtpDirectoryReader (const FtpDirectoryReader&) = delete; + FtpDirectoryReader& operator=(const FtpDirectoryReader&) = delete; + + static std::vector<FtpItem> parseMlsd(const std::string& buf, ServerEncoding enc) //throw SysError + { + std::vector<FtpItem> output; + for (const std::string& line : splitFtpResponse(buf)) + { + const FtpItem item = parseMlstLine(line, enc); //throw SysError + if (item.itemName == Zstr(".") || + item.itemName == Zstr("..")) + continue; + + output.push_back(item); + } + return output; + } + + static FtpItem parseMlstLine(const std::string& rawLine, ServerEncoding enc) //throw SysError + { + /* https://tools.ietf.org/html/rfc3659 + type=cdir;sizd=4096;modify=20170116230740;UNIX.mode=0755;UNIX.uid=874;UNIX.gid=869;unique=902g36e1c55; . + type=pdir;sizd=4096;modify=20170116230740;UNIX.mode=0755;UNIX.uid=874;UNIX.gid=869;unique=902g36e1c55; .. + type=file;size=4;modify=20170113063314;UNIX.mode=0600;UNIX.uid=874;UNIX.gid=869;unique=902g36e1c5d; readme.txt + type=dir;sizd=4096;modify=20170117144634;UNIX.mode=0755;UNIX.uid=874;UNIX.gid=869;unique=902g36e418a; folder + */ + FtpItem item; + + auto itBegin = rawLine.begin(); + if (startsWith(rawLine, ' ')) //leading blank is already trimmed if MLSD was processed by curl + ++itBegin; + auto itBlank = std::find(itBegin, rawLine.end(), ' '); + if (itBlank == rawLine.end()) + throw SysError(L"Item name not available. (" + utfTo<std::wstring>(rawLine) + L")"); + + const std::string facts(itBegin, itBlank); + item.itemName = serverToUtfEncoding(std::string(itBlank + 1, rawLine.end()), enc); //throw SysError + + std::string typeFact; + std::optional<uint64_t> fileSize; + + for (const std::string& fact : split(facts, ';', SplitType::SKIP_EMPTY)) + if (startsWithAsciiNoCase(fact, "type=")) //must be case-insensitive!!! + { + const std::string tmp = afterFirst(fact, '=', IF_MISSING_RETURN_NONE); + typeFact = beforeFirst(tmp, ':', IF_MISSING_RETURN_ALL); + } + else if (startsWithAsciiNoCase(fact, "size=")) + fileSize = stringTo<uint64_t>(afterFirst(fact, '=', IF_MISSING_RETURN_NONE)); + else if (startsWithAsciiNoCase(fact, "modify=")) + { + std::string modifyFact = afterFirst(fact, '=', IF_MISSING_RETURN_NONE); + modifyFact = beforeLast(modifyFact, '.', IF_MISSING_RETURN_ALL); //truncate millisecond precision if available + + const TimeComp tc = parseTime("%Y%m%d%H%M%S", modifyFact); + if (tc == TimeComp()) + throw SysError(L"Modification time could not be parsed. (" + utfTo<std::wstring>(modifyFact) + L")"); + + time_t utcTime = utcToTimeT(tc); //returns -1 on error + if (utcTime == -1) + { + if (tc.year == 1600 || //FTP on Windows phone: zero-initialized FILETIME equals "December 31, 1600" or "January 1, 1601" + tc.year == 1601) // => is this also relevant in this context of MLST UTC time?? + utcTime = 0; + else + throw SysError(L"Modification time could not be parsed. (" + utfTo<std::wstring>(modifyFact) + L")"); + } + item.modTime = utcTime; + } + + if (equalAsciiNoCase(typeFact, "cdir")) + return { AFS::ItemType::FOLDER, Zstr("."), 0, 0 }; + if (equalAsciiNoCase(typeFact, "pdir")) + return { AFS::ItemType::FOLDER, Zstr(".."), 0, 0 }; + + if (equalAsciiNoCase(typeFact, "dir")) + item.type = AFS::ItemType::FOLDER; + else if (equalAsciiNoCase(typeFact, "OS.unix=slink") || //the OS.unix=slink:/target syntax is a hack and often skips + equalAsciiNoCase(typeFact, "OS.unix=symlink")) //the target path after the colon: http://www.proftpd.org/docs/modules/mod_facts.html + item.type = AFS::ItemType::SYMLINK; + //It may be a good idea to NOT check for type "file" explicitly: see comment in native.cpp + + //evaluate parsing errors right now (+ report raw entry in error message!) + if (item.itemName.empty()) + throw SysError(L"Item name not available. (" + utfTo<std::wstring>(rawLine) + L")"); + + if (item.type == AFS::ItemType::FILE) + { + if (!fileSize) + throw SysError(L"File size not available. (" + utfTo<std::wstring>(rawLine) + L")"); + item.fileSize = *fileSize; + } + + //note: as far as the RFC goes, the "unique" fact is not required to act like a persistent file id! + return item; + } + + static std::vector<FtpItem> parseUnknown(const std::string& buf, ServerEncoding enc) //throw SysError + { + if (!buf.empty() && isDigit(buf[0])) //lame test to distinguish Unix/Dos formats as internally used by libcurl + return parseWindows(buf, enc); //throw SysError + return parseUnix(buf, enc); // + } + + //"ls -l" + static std::vector<FtpItem> parseUnix(const std::string& buf, ServerEncoding enc) //throw SysError + { + const std::vector<std::string> lines = splitFtpResponse(buf); + auto it = lines.begin(); + + if (it != lines.end() && startsWith(*it, "total ")) + ++it; + + const time_t utcTimeNow = std::time(nullptr); + const TimeComp tc = getUtcTime(utcTimeNow); + if (tc == TimeComp()) + throw SysError(L"Failed to determine current time: " + numberTo<std::wstring>(utcTimeNow)); + + const int utcCurrentYear = tc.year; + + std::optional<bool> unixListingHaveGroup_; //different listing format: better store at session level!? + std::vector<FtpItem> output; + + for (; it != lines.end(); ++it) + { + //unix listing without group: https://freefilesync.org/forum/viewtopic.php?t=4306 + if (!unixListingHaveGroup_) + unixListingHaveGroup_ = [&] + { + try + { + parseUnixLine(*it, utcTimeNow, utcCurrentYear, true /*haveGroup*/, enc); //throw SysError + return true; + } + catch (SysError&) + { + try + { + parseUnixLine(*it, utcTimeNow, utcCurrentYear, false /*haveGroup*/, enc); //throw SysError + return false; + } + catch (SysError&) {} + throw; + } + }(); + + const FtpItem item = parseUnixLine(*it, utcTimeNow, utcCurrentYear, *unixListingHaveGroup_, enc); //throw SysError + if (item.itemName == Zstr(".") || + item.itemName == Zstr("..")) + continue; + + output.push_back(item); + } + + return output; + } + + static FtpItem parseUnixLine(const std::string& rawLine, time_t utcTimeNow, int utcCurrentYear, bool haveGroup, ServerEncoding enc) //throw SysError + { + try + { + FtpLineParser parser(rawLine); + /* + total 4953 <- optional first line + drwxr-xr-x 1 root root 4096 Jan 10 11:58 version + -rwxr-xr-x 1 root root 1084 Sep 2 01:17 Unit Test.vcxproj.user + -rwxr-xr-x 1 1000 300 2217 Feb 28 2016 win32.manifest + lrwxr-xr-x 1 root root 18 Apr 26 15:17 Projects -> /mnt/hgfs/Projects + + file type: -:file l:symlink d:directory b:block device p:named pipe c:char device s:socket + + permissions: (r|-)(w|-)(x|s|S|-) user + (r|-)(w|-)(x|s|S|-) group s := S + x S = Setgid + (r|-)(w|-)(x|t|T|-) others t := T + x T = sticky bit + + Alternative formats: + Unix, no group ("ls -alG") https://freefilesync.org/forum/viewtopic.php?t=4306 + dr-xr-xr-x 2 root 512 Apr 8 1994 etc + + Yet to be seen in the wild: + Netware: + d [R----F--] supervisor 512 Jan 16 18:53 login + - [R----F--] rhesus 214059 Oct 20 15:27 cx.exe + + NetPresenz for the Mac: + -------r-- 326 1391972 1392298 Nov 22 1995 MegaPhone.sit + drwxrwxr-x folder 2 May 10 1996 network + */ + const std::string typeTag = parser.readRange(1, [](char c) //throw SysError + { + return c == '-' || c == 'b' || c == 'c' || c == 'd' || c == 'l' || c == 'p' || c == 's'; + }); + //------------------------------------------------------------------------------------ + //permissions + parser.readRange(9, [](char c) //throw SysError + { + return c == '-' || c == 'r' || c == 'w' || c == 'x' || c == 's' || c == 'S' || c == 't' || c == 'T'; + }); + parser.readRange(&isWhiteSpace<char>); //throw SysError + //------------------------------------------------------------------------------------ + //hard-link count (no separators) + parser.readRange(&isDigit<char>); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + //------------------------------------------------------------------------------------ + //user + parser.readRange(std::not_fn(isWhiteSpace<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + //------------------------------------------------------------------------------------ + //group + if (haveGroup) + { + parser.readRange(std::not_fn(isWhiteSpace<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + } + //------------------------------------------------------------------------------------ + //file size (no separators) + const uint64_t fileSize = stringTo<uint64_t>(parser.readRange(&isDigit<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + //------------------------------------------------------------------------------------ + const std::string monthStr = parser.readRange(std::not_fn(isWhiteSpace<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + + const char* months[] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; + auto itMonth = std::find_if(std::begin(months), std::end(months), [&](const char* name) { return equalAsciiNoCase(name, monthStr); }); + if (itMonth == std::end(months)) + throw SysError(L"Unknown month name."); + //------------------------------------------------------------------------------------ + const int day = stringTo<int>(parser.readRange(&isDigit<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + if (day < 1 || day > 31) + throw SysError(L"Unexpected day of month."); + //------------------------------------------------------------------------------------ + const std::string timeOrYear = parser.readRange([](char c) { return c == ':' || isDigit(c); }); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + + TimeComp timeComp; + timeComp.month = 1 + static_cast<int>(itMonth - std::begin(months)); + timeComp.day = day; + + if (contains(timeOrYear, ':')) + { + const int hour = stringTo<int>(beforeFirst(timeOrYear, ':', IF_MISSING_RETURN_NONE)); + const int minute = stringTo<int>(afterFirst (timeOrYear, ':', IF_MISSING_RETURN_NONE)); + if (hour < 0 || hour > 23 || minute < 0 || minute > 59) + throw SysError(L"Failed to parse file time."); + + timeComp.hour = hour; + timeComp.minute = minute; + timeComp.year = utcCurrentYear; //tentatively + const time_t serverLocalTime = utcToTimeT(timeComp); //returns -1 on error + if (serverLocalTime == -1) + throw SysError(L"Modification time could not be parsed."); + + if (serverLocalTime - utcTimeNow > 3600 * 24) //time-zones range from UTC-12:00 to UTC+14:00, consider DST; FileZilla uses 1 day tolerance + --timeComp.year; //"more likely" this time is from last year + } + else if (timeOrYear.size() == 4) + { + timeComp.year = stringTo<int>(timeOrYear); + + if (timeComp.year < 1600 || timeComp.year > utcCurrentYear + 1 /*leeway*/) + throw SysError(L"Failed to parse file time."); + } + else + throw SysError(L"Failed to parse file time."); + + //let's pretend the time listing is UTC (same behavior as FileZilla): hopefully MLSD will make this mess obsolete soon... + time_t utcTime = utcToTimeT(timeComp); //returns -1 on error + if (utcTime == -1) + { + if (timeComp.year == 1600 || //FTP on Windows phone: zero-initialized FILETIME equals "December 31, 1600" or "January 1, 1601" + timeComp.year == 1601) // + utcTime = 0; + else + throw SysError(L"Modification time could not be parsed."); + } + //------------------------------------------------------------------------------------ + const std::string trail = parser.readRange([](char) { return true; }); //throw SysError + std::string itemName; + if (typeTag == "l") + itemName = beforeFirst(trail, " -> ", IF_MISSING_RETURN_NONE); + else + itemName = trail; + if (itemName.empty()) + throw SysError(L"Item name not available."); + + if (itemName == "." || itemName == "..") //sometimes returned, e.g. by freefilesync.org + return { AFS::ItemType::FOLDER, utfTo<Zstring>(itemName), 0, 0 }; + //------------------------------------------------------------------------------------ + FtpItem item; + if (typeTag == "d") + item.type = AFS::ItemType::FOLDER; + else if (typeTag == "l") + item.type = AFS::ItemType::SYMLINK; + else + item.fileSize = fileSize; + + item.itemName = serverToUtfEncoding(itemName, enc); //throw SysError + item.modTime = utcTime; + + return item; + } + catch (const SysError& e) + { + throw SysError(L"Failed to parse FTP response. (" + utfTo<std::wstring>(rawLine) + L")" + (haveGroup ? L"" : L" [no-group]") + L" " + e.toString()); + } + } + + + //"dir" + static std::vector<FtpItem> parseWindows(const std::string& buf, ServerEncoding enc) //throw SysError + { + /* + Test server: test.rebex.net username:demo pw:password useSsl = true + + listing supported by libcurl (US server) + 10-27-15 03:46AM <DIR> pub + 04-08-14 03:09PM 11,399 readme.txt + + Datalogic Windows CE 5.0 + 01-01-98 13:00 <DIR> Storage Card + + IIS option "four-digit years" + 06-22-2017 04:25PM <DIR> test + 06-20-2017 12:50PM 1875499 zstring.obj + + Alternative formats (yet to be seen in the wild) + "dir" on Windows, US: + 10/27/2015 03:46 AM <DIR> pub + 04/08/2014 03:09 PM 11,399 readme.txt + + "dir" on Windows, German: + 21.09.2016 18:31 <DIR> Favorites + 12.01.2017 19:57 11.399 gsview64.ini + */ + const TimeComp tc = getUtcTime(); + if (tc == TimeComp()) + throw SysError(L"Failed to determine current time: " + numberTo<std::wstring>(std::time(nullptr))); + const int utcCurrentYear = tc.year; + + std::vector<FtpItem> output; + for (const std::string& line : splitFtpResponse(buf)) + { + try + { + FtpLineParser parser(line); + + const int month = stringTo<int>(parser.readRange(2, &isDigit<char>)); //throw SysError + parser.readRange(1, [](char c) { return c == '-' || c == '/'; }); //throw SysError + const int day = stringTo<int>(parser.readRange(2, &isDigit<char>)); //throw SysError + parser.readRange(1, [](char c) { return c == '-' || c == '/'; }); //throw SysError + const std::string yearString = parser.readRange(&isDigit<char>); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + + if (month < 1 || month > 12 || day < 1 || day > 31) + throw SysError(L"Failed to parse file time."); + + int year = 0; + if (yearString.size() == 2) + { + year = (utcCurrentYear / 100) * 100 + stringTo<int>(yearString); + if (year > utcCurrentYear + 1 /*local time leeway*/) + year -= 100; + } + else if (yearString.size() == 4) + year = stringTo<int>(yearString); + else + throw SysError(L"Failed to parse file time."); + //------------------------------------------------------------------------------------ + int hour = stringTo<int>(parser.readRange(2, &isDigit<char>)); //throw SysError + parser.readRange(1, [](char c) { return c == ':'; }); //throw SysError + const int minute = stringTo<int>(parser.readRange(2, &isDigit<char>)); //throw SysError + if (!isWhiteSpace(parser.peekNextChar())) + { + const std::string period = parser.readRange(2, [](char c) { return c == 'A' || c == 'P' || c == 'M'; }); //throw SysError + if (period == "PM") + { + if (0 <= hour && hour < 12) + hour += 12; + } + else if (hour == 12) + hour = 0; + } + parser.readRange(&isWhiteSpace<char>); //throw SysError + + if (hour < 0 || hour > 23 || minute < 0 || minute > 59) + throw SysError(L"Failed to parse file time."); + //------------------------------------------------------------------------------------ + TimeComp timeComp; + timeComp.year = year; + timeComp.month = month; + timeComp.day = day; + timeComp.hour = hour; + timeComp.minute = minute; + //let's pretend the time listing is UTC (same behavior as FileZilla): hopefully MLSD will make this mess obsolete soon... + time_t utcTime = utcToTimeT(timeComp); //returns -1 on error + if (utcTime == -1) + { + if (timeComp.year == 1600 || //FTP on Windows phone: zero-initialized FILETIME equals "December 31, 1600" or "January 1, 1601" + timeComp.year == 1601) // + utcTime = 0; + else + throw SysError(L"Modification time could not be parsed."); + } + //------------------------------------------------------------------------------------ + const std::string dirTagOrSize = parser.readRange(std::not_fn(isWhiteSpace<char>)); //throw SysError + parser.readRange(&isWhiteSpace<char>); //throw SysError + + const bool isDir = dirTagOrSize == "<DIR>"; + uint64_t fileSize = 0; + if (!isDir) + { + std::string sizeStr = dirTagOrSize; + replace(sizeStr, ',', ""); + replace(sizeStr, '.', ""); + if (!std::all_of(sizeStr.begin(), sizeStr.end(), &isDigit<char>)) + throw SysError(L"Failed to parse file size."); + fileSize = stringTo<uint64_t>(sizeStr); + } + //------------------------------------------------------------------------------------ + const std::string itemName = parser.readRange([](char) { return true; }); //throw SysError + if (itemName.empty()) + throw SysError(L"Folder contains child item without a name."); + + if (itemName == "." || itemName == "..") + continue; + //------------------------------------------------------------------------------------ + FtpItem item; + if (isDir) + item.type = AFS::ItemType::FOLDER; + item.itemName = serverToUtfEncoding(itemName, enc); //throw SysError + item.fileSize = fileSize; + item.modTime = utcTime; + + output.push_back(item); + } + catch (const SysError& e) + { + throw SysError(L"Failed to parse FTP response. (" + utfTo<std::wstring>(line) + L") " + e.toString()); + } + } + + return output; + } +}; + + +class SingleFolderTraverser +{ +public: + SingleFolderTraverser(const FtpLoginInfo& login, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/) + : workload_(workload), login_(login) + { + while (!workload_.empty()) + { + auto wi = std::move(workload_. back()); //yes, no strong exception guarantee (std::bad_alloc) + /**/ workload_.pop_back(); // + const auto& [folderPath, cb] = wi; + + tryReportingDirError([&] //throw X + { + traverseWithException(folderPath, *cb); //throw FileError, X + }, *cb); + } + } + +private: + SingleFolderTraverser (const SingleFolderTraverser&) = delete; + SingleFolderTraverser& operator=(const SingleFolderTraverser&) = delete; + + void traverseWithException(const AfsPath& dirPath, AFS::TraverserCallback& cb) //throw FileError, X + { + for (const FtpItem& item : FtpDirectoryReader::execute(login_, dirPath)) //throw FileError + { + const AfsPath itemPath(nativeAppendPaths(dirPath.value, item.itemName)); + + switch (item.type) + { + case AFS::ItemType::FILE: + cb.onFile({ item.itemName, item.fileSize, item.modTime, AFS::FileId(), nullptr /*symlinkInfo*/ }); //throw X + break; + + case AFS::ItemType::FOLDER: + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ item.itemName, nullptr /*symlinkInfo*/ })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + break; + + case AFS::ItemType::SYMLINK: + { + const AFS::SymlinkInfo linkInfo = { item.itemName, item.modTime }; + switch (cb.onSymlink(linkInfo)) //throw X + { + case AFS::TraverserCallback::LINK_FOLLOW: + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ item.itemName, &linkInfo })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + break; + + case AFS::TraverserCallback::LINK_SKIP: + break; + } + } + break; + } + } + } + + std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>> workload_; + const FtpLoginInfo login_; +}; + + +void traverseFolderRecursiveFTP(const FtpLoginInfo& login, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/, size_t) //throw X +{ + SingleFolderTraverser dummy(login, workload); //throw X +} +//=========================================================================================================================== +//=========================================================================================================================== + +void ftpFileDownload(const FtpLoginInfo& login, const AfsPath& afsFilePath, //throw FileError, X + const std::function<void(const void* buffer, size_t bytesToWrite)>& writeBlock /*throw X*/) +{ + std::exception_ptr exception; + + auto onBytesReceived = [&](const void* buffer, size_t len) + { + try + { + writeBlock(buffer, len); //throw X + return len; + } + catch (...) + { + exception = std::current_exception(); + return len + 1; //signal error condition => CURLE_WRITE_ERROR + } + }; + + using CbType = decltype(onBytesReceived); + using CbWrapperType = size_t (*)(const void* buffer, size_t size, size_t nitems, void* callbackData); //needed for cdecl function pointer cast + CbWrapperType onBytesReceivedWrapper = [](const void* buffer, size_t size, size_t nitems, void* callbackData) + { + auto cb = static_cast<CbType*>(callbackData); //free this poor little C-API from its shackles and redirect to a proper lambda + return (*cb)(buffer, size * nitems); + }; + + accessFtpSession(login, [&](FtpSession& session) //throw FileError + { + try + { + session.perform(&afsFilePath, false /*isDir*/, //throw FileError, SysError + { + FtpSession::Option(CURLOPT_WRITEDATA, &onBytesReceived), + FtpSession::Option(CURLOPT_WRITEFUNCTION, onBytesReceivedWrapper), + }, true /*requiresUtf8*/, login.timeoutSec); + } + catch (const SysError& e) + { + if (exception) + std::rethrow_exception(exception); + + throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(getCurlDisplayPath(login.server, afsFilePath))), e.toString()); + } + }); +} + + +/* +File already existing: + freefilesync.org: overwrites + FileZilla Server: overwrites + Windows IIS: overwrites +*/ +void ftpFileUpload(const FtpLoginInfo& login, const AfsPath& afsFilePath, //throw FileError, X + const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/) //returning 0 signals EOF: Posix read() semantics +{ + std::exception_ptr exception; + + auto getBytesToSend = [&](void* buffer, size_t len) -> size_t + { + try + { + //libcurl calls back until 0 bytes are returned (Posix read() semantics), or, + //if CURLOPT_INFILESIZE_LARGE was set, after exactly this amount of bytes + const size_t bytesRead = readBlock(buffer, len);//throw X; return "bytesToRead" bytes unless end of stream! + return bytesRead; + } + catch (...) + { + exception = std::current_exception(); + return CURL_READFUNC_ABORT; //signal error condition => CURLE_ABORTED_BY_CALLBACK + } + }; + + using CbType = decltype(getBytesToSend); + using CbWrapperType = size_t (*)(void* buffer, size_t size, size_t nitems, void* callbackData); + CbWrapperType getBytesToSendWrapper = [](void* buffer, size_t size, size_t nitems, void* callbackData) + { + auto cb = static_cast<CbType*>(callbackData); //free this poor little C-API from its shackles and redirect to a proper lambda + return (*cb)(buffer, size * nitems); + }; + + accessFtpSession(login, [&](FtpSession& session) //throw FileError + { + try + { + /* + struct curl_slist* quote = nullptr; + ZEN_ON_SCOPE_EXIT(::curl_slist_free_all(quote)); + + //"prefix the command with an asterisk to make libcurl continue even if the command fails" + quote = ::curl_slist_append(quote, ("*DELE " + session.getServerRelPathInternal(afsFilePath)).c_str()); //throw FileError + + //optimize fail-safe copy with RNFR/RNTO as CURLOPT_POSTQUOTE? -> even slightly *slower* than RNFR/RNTO as additional curl_easy_perform() + */ + session.perform(&afsFilePath, false /*isDir*/, //throw FileError, SysError + { + FtpSession::Option(CURLOPT_UPLOAD, 1L), + //FtpSession::Option(CURLOPT_INFILESIZE_LARGE, static_cast<curl_off_t>(inputBuffer.size())), + //=> CURLOPT_INFILESIZE_LARGE does not issue a specific FTP command, but is used by libcurl only! + + FtpSession::Option(CURLOPT_READDATA, &getBytesToSend), + FtpSession::Option(CURLOPT_READFUNCTION, getBytesToSendWrapper), + + //FtpSession::Option(CURLOPT_PREQUOTE, quote), + //FtpSession::Option(CURLOPT_POSTQUOTE, quote), + }, true /*requiresUtf8*/, login.timeoutSec); + } + catch (const SysError& e) + { + if (exception) + std::rethrow_exception(exception); + + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getCurlDisplayPath(login.server, afsFilePath))), e.toString()); + } + }); +} + +//=========================================================================================================================== + +struct InputStreamFtp : public AbstractFileSystem::InputStream +{ + InputStreamFtp(const FtpLoginInfo& login, + const AfsPath& afsPath, + const IOCallback& notifyUnbufferedIO /*throw X*/) : + notifyUnbufferedIO_(notifyUnbufferedIO) + { + worker_ = InterruptibleThread([asyncStreamOut = this->asyncStreamIn_, login, afsPath] + { + setCurrentThreadName(("Istream[FTP] " + utfTo<std::string>(getCurlDisplayPath(login.server, afsPath))). c_str()); + try + { + auto writeBlock = [&](const void* buffer, size_t bytesToWrite) + { + return asyncStreamOut->write(buffer, bytesToWrite); //throw ThreadInterruption + }; + ftpFileDownload(login, afsPath, writeBlock); //throw FileError, ThreadInterruption + + asyncStreamOut->closeStream(); + } + catch (FileError&) { asyncStreamOut->setWriteError(std::current_exception()); } //let ThreadInterruption pass through! + }); + } + + ~InputStreamFtp() + { + asyncStreamIn_->setReadError(std::make_exception_ptr(ThreadInterruption())); + worker_.join(); + } + + size_t read(void* buffer, size_t bytesToRead) override //throw FileError, (ErrorFileLocked), X; return "bytesToRead" bytes unless end of stream! + { + const size_t bytesRead = asyncStreamIn_->read(buffer, bytesToRead); //throw FileError + reportBytesProcessed(); //throw X + return bytesRead; + //no need for asyncStreamIn_->checkWriteErrors(): once end of stream is reached, asyncStreamOut->closeStream() was called => no errors occured + } + + size_t getBlockSize() const override { return 64 * 1024; } //non-zero block size is AFS contract! + + std::optional<AFS::StreamAttributes> getAttributesBuffered() override //throw FileError + { + return {}; //there is no stream handle => no buffered attribute access! + //PERF: get attributes during file download? + // CURLOPT_FILETIME: test case 77 files, 4MB: overall copy time increases by 12% + // CURLOPT_PREQUOTE/CURLOPT_PREQUOTE/CURLOPT_POSTQUOTE + MDTM: test case 77 files, 4MB: overall copy time increases by 12% + } + +private: + void reportBytesProcessed() //throw X + { + const int64_t totalBytesDownloaded = asyncStreamIn_->getTotalBytesWritten(); + if (notifyUnbufferedIO_) notifyUnbufferedIO_(totalBytesDownloaded - totalBytesReported_); //throw X + totalBytesReported_ = totalBytesDownloaded; + } + + const IOCallback notifyUnbufferedIO_; //throw X + int64_t totalBytesReported_ = 0; + std::shared_ptr<AsyncStreamBuffer> asyncStreamIn_ = std::make_shared<AsyncStreamBuffer>(FTP_STREAM_BUFFER_SIZE); + InterruptibleThread worker_; +}; + +//=========================================================================================================================== + +struct OutputStreamFtp : public AbstractFileSystem::OutputStreamImpl +{ + OutputStreamFtp(const FtpLoginInfo& login, + const AfsPath& afsPath, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) : + login_(login), + afsPath_(afsPath), + modTime_(modTime), + notifyUnbufferedIO_(notifyUnbufferedIO) + { + worker_ = InterruptibleThread([asyncStreamIn = this->asyncStreamOut_, login, afsPath] + { + setCurrentThreadName(("Ostream[FTP] " + utfTo<std::string>(getCurlDisplayPath(login.server, afsPath))). c_str()); + try + { + auto readBlock = [&](void* buffer, size_t bytesToRead) + { + //returns "bytesToRead" bytes unless end of stream! => maps nicely into Posix read() semantics expected by ftpFileUpload() + return asyncStreamIn->read(buffer, bytesToRead); //throw ThreadInterruption + }; + ftpFileUpload(login, afsPath, readBlock); //throw FileError, ThreadInterruption + assert(asyncStreamIn->getTotalBytesRead() == asyncStreamIn->getTotalBytesWritten()); + } + catch (FileError&) { asyncStreamIn->setReadError(std::current_exception()); } //let ThreadInterruption pass through! + }); + } + + ~OutputStreamFtp() + { + if (worker_.joinable()) + { + asyncStreamOut_->setWriteError(std::make_exception_ptr(ThreadInterruption())); + worker_.join(); + } + } + + void write(const void* buffer, size_t bytesToWrite) override //throw FileError, X + { + asyncStreamOut_->write(buffer, bytesToWrite); //throw FileError + reportBytesProcessed(); //throw X + } + + AFS::FinalizeResult finalize() override //throw FileError, X + { + asyncStreamOut_->closeStream(); + + while (!worker_.tryJoinFor(std::chrono::milliseconds(50))) + reportBytesProcessed(); //throw X + reportBytesProcessed(); //[!] once more, now that *all* bytes were written + + asyncStreamOut_->checkReadErrors(); //throw FileError + //-------------------------------------------------------------------- + + AFS::FinalizeResult result; + //result.fileId = ... -> not supported by FTP + try + { + setModTimeIfAvailable(); //throw FileError, follows symlinks + /* is setting modtime after closing the file handle a pessimization? + FTP: no: could set modtime via CURLOPT_POSTQUOTE (but this would internally trigger an extra round-trip anyway!) */ + } + catch (const FileError& e) { result.errorModTime = FileError(e.toString()); /*avoid slicing*/ } + + return result; + } + +private: + void reportBytesProcessed() //throw X + { + const int64_t totalBytesUploaded = asyncStreamOut_->getTotalBytesRead(); + if (notifyUnbufferedIO_) notifyUnbufferedIO_(totalBytesUploaded - totalBytesReported_); //throw X + totalBytesReported_ = totalBytesUploaded; + } + + void setModTimeIfAvailable() const //throw FileError, follows symlinks + { + assert(!worker_.joinable()); + if (modTime_) + try + { + const std::string isoTime = formatTime<std::string>("%Y%m%d%H%M%S", getUtcTime(*modTime_)); //returns empty string on failure + if (isoTime.empty()) + throw SysError(L"Invalid modification time (time_t: " + numberTo<std::wstring>(*modTime_) + L")"); + + accessFtpSession(login_, [&](FtpSession& session) //throw FileError + { + if (!session.supportsMfmt(login_.timeoutSec)) //throw FileError + throw SysError(L"Server does not support the MFMT command."); + + session.runSingleFtpCommand("MFMT " + isoTime + " " + session.getServerRelPathInternal(afsPath_, login_.timeoutSec), + true /*requiresUtf8*/, login_.timeoutSec); //throw FileError, SysError + //Does MFMT follow symlinks?? Anyway, our FTP implementation supports folder symlinks only + }); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot write modification time of %x."), L"%x", fmtPath(getCurlDisplayPath(login_.server, afsPath_))), e.toString()); + } + } + + const FtpLoginInfo login_; + const AfsPath afsPath_; + const std::optional<time_t> modTime_; + const IOCallback notifyUnbufferedIO_; //throw X + int64_t totalBytesReported_ = 0; + std::shared_ptr<AsyncStreamBuffer> asyncStreamOut_ = std::make_shared<AsyncStreamBuffer>(FTP_STREAM_BUFFER_SIZE); + InterruptibleThread worker_; +}; + +//--------------------------------------------------------------------------------------------------------------------------- +//=========================================================================================================================== + +class FtpFileSystem : public AbstractFileSystem +{ +public: + FtpFileSystem(const FtpLoginInfo& login) : login_(login) {} + +private: + Zstring getInitPathPhrase(const AfsPath& afsPath) const override { return concatenateFtpFolderPathPhrase(login_, afsPath); } + + std::wstring getDisplayPath(const AfsPath& afsPath) const override { return getCurlDisplayPath(login_.server, afsPath); } + + bool isNullFileSystem() const override { return login_.server.empty(); } + + int compareDeviceSameAfsType(const AbstractFileSystem& afsRhs) const override + { + const FtpLoginInfo& lhs = login_; + const FtpLoginInfo& rhs = static_cast<const FtpFileSystem&>(afsRhs).login_; + + //exactly the type of case insensitive comparison we need for server names! + const int rv = compareAsciiNoCase(lhs.server, rhs.server); //https://msdn.microsoft.com/en-us/library/windows/desktop/ms738519#IDNs + if (rv != 0) + return rv; + + //port does NOT create a *different* data source!!! -> same thing for password! + + //username: usually *does* create different folder view for FTP + return compareString(lhs.username, rhs.username); //case sensitive! + } + + //---------------------------------------------------------------------------------------------------------------- + + ItemType getItemType(const AfsPath& afsPath) const override //throw FileError + { + //don't use MLST: broken for Pure-FTPd: https://freefilesync.org/forum/viewtopic.php?t=4287 + + const std::optional<AfsPath> parentAfsPath = getParentPath(afsPath); + if (!parentAfsPath) //device root => quick access tests: just see if the server responds at all! + { + //don't use PWD: if last access deleted the working dir, PWD will fail on some servers, e.g. https://freefilesync.org/forum/viewtopic.php?t=4314 + //FEAT: are there servers that don't support this command? fuck, yes: "550 FEAT: Operation not permitted" => buggy server not granting access, despite support! + //=> but "HELP", and "NOOP" work, right?? https://en.wikipedia.org/wiki/List_of_FTP_commands + //Fuck my life: even "HELP" is not always implemented: https://freefilesync.org/forum/viewtopic.php?t=6002 + //Screw this, just traverse the root folder: (only a single round-trip for FTP) + FtpDirectoryReader::execute(login_, afsPath); //throw FileError + return ItemType::FOLDER; + } + + const Zstring itemName = getItemName(afsPath); + assert(!itemName.empty()); + try + { + //is the underlying file system case-sensitive? we don't know => assume "case-sensitive" + //=> all path parts (except the base folder part!) can be expected to have the right case anyway after traversal + traverseFolderFlat(*parentAfsPath, //throw FileError + [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, + [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, + [&](const SymlinkInfo& si) { if (si.itemName == itemName) throw ItemType::SYMLINK; }); + } + catch (const ItemType& type) { return type; } //yes, exceptions for control-flow are bad design... but, but... + + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"File not found."); + } + + std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const override //throw FileError + { + const std::optional<AfsPath> parentAfsPath = getParentPath(afsPath); + if (!parentAfsPath) //device root + return getItemType(afsPath); //throw FileError; do a simple access test + + const Zstring itemName = getItemName(afsPath); + assert(!itemName.empty()); + try + { + traverseFolderFlat(*parentAfsPath, //throw FileError + [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, + [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, + [&](const SymlinkInfo& si) { if (si.itemName == itemName) throw ItemType::SYMLINK; }); + } + catch (const ItemType& type) { return type; } //yes, exceptions for control-flow are bad design... but, but... + catch (FileError&) + { + const std::optional<ItemType> parentType = itemStillExists(*parentAfsPath); //throw FileError + if (parentType && *parentType != ItemType::FILE) //obscure, but possible (and not an error) + throw; //parent path existing, so traversal should not have failed! + } + return {}; + } + //---------------------------------------------------------------------------------------------------------------- + + //already existing: fail/ignore + //=> FTP will (most likely) fail and give a clear error message: + // freefilesync.org: "550 Can't create directory: File exists" + // FileZilla Server: "550 Directory already exists" + // Windows IIS: "550 Cannot create a file when that file already exists" + void createFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + accessFtpSession(login_, [&](FtpSession& session) //throw FileError + { + session.runSingleFtpCommand("MKD " + session.getServerRelPathInternal(afsPath, login_.timeoutSec), + true /*requiresUtf8*/, login_.timeoutSec); //throw FileError, SysError + }); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot create directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeFilePlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + accessFtpSession(login_, [&](FtpSession& session) //throw FileError + { + session.runSingleFtpCommand("DELE " + session.getServerRelPathInternal(afsPath, login_.timeoutSec), + true /*requiresUtf8*/, login_.timeoutSec); //throw FileError, SysError + }); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot delete file %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeSymlinkPlain(const AfsPath& afsPath) const override //throw FileError + { + this->removeFilePlain(afsPath); //throw FileError + //works fine for Linux hosts, but what about Windows-hosted FTP??? Distinguish DELE/RMD? + //Windows test, FileZilla Server and Windows IIS FTP: all symlinks are reported as regular folders + } + + void removeFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + accessFtpSession(login_, [&](FtpSession& session) //throw FileError + { + session.runSingleFtpCommand("RMD " + session.getServerRelPathInternal(afsPath, login_.timeoutSec), + true /*requiresUtf8*/, login_.timeoutSec); //throw FileError, SysError + }); + } + catch (const SysError& e) + { + //tested freefilesync.org: RMD will fail for symlinks! + bool symlinkExists = false; + try { symlinkExists = getItemType(afsPath) == ItemType::SYMLINK; } /*throw FileError*/ catch (FileError&) {} //previous exception is more relevant + + if (symlinkExists) + return removeSymlinkPlain(afsPath); //throw FileError + + throw FileError(replaceCpy(_("Cannot delete directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + //Windows test, FileZilla Server and Windows IIS FTP: all symlinks are reported as regular folders + } + + void removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const override //one call for each object! + { + //default implementation: folder traversal + AbstractFileSystem::removeFolderIfExistsRecursion(afsPath, onBeforeFileDeletion, onBeforeFolderDeletion); //throw FileError, X + } + + //---------------------------------------------------------------------------------------------------------------- + AbstractPath getSymlinkResolvedPath(const AfsPath& afsPath) const override //throw FileError + { + throw FileError(replaceCpy(_("Cannot determine final path for %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Reading symlink content not supported for FTP devices."); + } + + std::string getSymlinkBinaryContent(const AfsPath& afsPath) const override //throw FileError + { + throw FileError(replaceCpy(_("Cannot resolve symbolic link %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Reading symlink content not supported for FTP devices."); + } + //---------------------------------------------------------------------------------------------------------------- + + //return value always bound: + std::unique_ptr<InputStream> getInputStream(const AfsPath& afsPath, const IOCallback& notifyUnbufferedIO /*throw X*/) const override //throw FileError, (ErrorFileLocked) + { + return std::make_unique<InputStreamFtp>(login_, afsPath, notifyUnbufferedIO); + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + //=> most FTP servers overwrite, but some (e.g. IIS) can be configured to fail, others (pureFTP) can be configured to auto-rename: + // https://download.pureftpd.org/pub/pure-ftpd/doc/README + // '-r': Never overwrite existing files. Uploading a file whose name already exists cause an automatic rename. Files are called xyz, xyz.1, xyz.2, xyz.3, etc. + std::unique_ptr<OutputStreamImpl> getOutputStream(const AfsPath& afsPath, //throw FileError + std::optional<uint64_t> streamSize, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + return std::make_unique<OutputStreamFtp>(login_, afsPath, modTime, notifyUnbufferedIO); + } + + //---------------------------------------------------------------------------------------------------------------- + void traverseFolderRecursive(const TraverserWorkload& workload /*throw X*/, size_t parallelOps) const override + { + traverseFolderRecursiveFTP(login_, workload, parallelOps); //throw X + } + //---------------------------------------------------------------------------------------------------------------- + + //symlink handling: follow link! + //target existing: undefined behavior! (fail/overwrite/auto-rename) + FileCopyResult copyFileForSameAfsType(const AfsPath& afsPathSource, const StreamAttributes& attrSource, //throw FileError, (ErrorFileLocked), X + const AbstractPath& apTarget, bool copyFilePermissions, const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + //no native FTP file copy => use stream-based file copy: + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for FTP devices."); + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + return copyFileAsStream(afsPathSource, attrSource, apTarget, notifyUnbufferedIO); //throw FileError, (ErrorFileLocked), X + } + + //target existing: fail/ignore + //symlink handling: follow link! + void copyNewFolderForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override //throw FileError + { + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for FTP devices."); + + //already existing: fail/ignore + AFS::createFolderPlain(apTarget); //throw FileError + } + + void copySymlinkForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override + { + throw FileError(replaceCpy(replaceCpy(_("Cannot copy symbolic link %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))), + L"Creating symlinks is not supported by libcurl."); + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + //=> most linux-based FTP servers overwrite, Windows-based servers fail (but most can be configured to behave differently) + // freefilesync.org: silent overwrite + // Windows IIS: CURLE_QUOTE_ERROR: QUOT command failed with 550 Cannot create a file when that file already exists. + // FileZilla Server: CURLE_QUOTE_ERROR: QUOT command failed with 553 file exists + void moveAndRenameItemForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget) const override //throw FileError, ErrorDifferentVolume + { + auto generateErrorMsg = [&] { return replaceCpy(replaceCpy(_("Cannot move file %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))); + }; + + if (compareDeviceSameAfsType(apTarget.afsDevice.ref()) != 0) + throw ErrorDifferentVolume(generateErrorMsg(), L"Different FTP volume."); + + try + { + accessFtpSession(login_, [&](FtpSession& session) //throw FileError + { + struct curl_slist* quote = nullptr; + ZEN_ON_SCOPE_EXIT(::curl_slist_free_all(quote)); + quote = ::curl_slist_append(quote, ("RNFR " + session.getServerRelPathInternal(afsPathSource, login_.timeoutSec)).c_str()); //throw FileError + quote = ::curl_slist_append(quote, ("RNTO " + session.getServerRelPathInternal(apTarget.afsPath, login_.timeoutSec)).c_str()); // + + session.perform(nullptr /*re-use last-used path*/, true /*isDir*/, //throw FileError, SysError + { + FtpSession::Option(CURLOPT_NOBODY, 1L), + FtpSession::Option(CURLOPT_QUOTE, quote), + }, true /*requiresUtf8*/, login_.timeoutSec); + }); + } + catch (const SysError& e) + { + throw FileError(generateErrorMsg(), e.toString()); + } + } + + bool supportsPermissions(const AfsPath& afsPath) const override { return false; } //throw FileError + //wait until there is real demand for copying from and to FTP with permissions => use stream-based file copy: + + //---------------------------------------------------------------------------------------------------------------- + ImageHolder getFileIcon (const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } //noexcept; optional return value + ImageHolder getThumbnailImage(const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } // + + void authenticateAccess(bool allowUserInteraction) const override {} //throw FileError + + int getAccessTimeout() const override { return login_.timeoutSec; } //returns "0" if no timeout in force + + bool hasNativeTransactionalCopy() const override { return false; } + //---------------------------------------------------------------------------------------------------------------- + + uint64_t getFreeDiskSpace(const AfsPath& afsPath) const override { return 0; } //throw FileError, returns 0 if not available + + bool supportsRecycleBin(const AfsPath& afsPath, const std::function<void ()>& onUpdateGui) const override { return false; } //throw FileError + + std::unique_ptr<RecycleSession> createRecyclerSession(const AfsPath& afsPath) const override //throw FileError, return value must be bound! + { + assert(false); //see supportsRecycleBin() + throw FileError(L"Recycle Bin not supported for FTP devices."); + } + + void recycleItemIfExists(const AfsPath& afsPath) const override //throw FileError + { + assert(false); //see supportsRecycleBin() + throw FileError(replaceCpy(_("Unable to move %x to the recycle bin."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Recycle Bin not supported for FTP devices."); + } + + const FtpLoginInfo login_; +}; + +//=========================================================================================================================== + +//expects "clean" login data, see condenseToFtpFolderPathPhrase() +Zstring concatenateFtpFolderPathPhrase(const FtpLoginInfo& login, const AfsPath& afsPath) //noexcept +{ + Zstring port; + if (login.port > 0) + port = Zstr(":") + numberTo<Zstring>(login.port); + + Zstring options; + if (login.timeoutSec != FtpLoginInfo().timeoutSec) + options += Zstr("|timeout=") + numberTo<Zstring>(login.timeoutSec); + + if (login.useSsl) + options += Zstr("|ssl"); + + if (!login.password.empty()) //password always last => visually truncated by folder input field + options += Zstr("|pass64=") + encodePasswordBase64(login.password); + + Zstring username; + if (!login.username.empty()) + username = encodeFtpUsername(login.username) + Zstr("@"); + + return Zstring(ftpPrefix) + Zstr("//") + username + login.server + port + getServerRelPath(afsPath) + options; +} +} + + +Zstring fff::condenseToFtpFolderPathPhrase(const FtpLoginInfo& login, const Zstring& relPath) //noexcept +{ + FtpLoginInfo loginTmp = login; + + //clean-up input: + trim(loginTmp.server); + trim(loginTmp.username); + + loginTmp.timeoutSec = std::max(1, loginTmp.timeoutSec); + + if (startsWithAsciiNoCase(loginTmp.server, Zstr("http:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("https:")) || + startsWithAsciiNoCase(loginTmp.server, Zstr("ftp:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("ftps:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("sftp:" ))) + loginTmp.server = afterFirst(loginTmp.server, Zstr(':'), IF_MISSING_RETURN_NONE); + trim(loginTmp.server, true, false, [](Zchar c) { return c == Zstr('/') || c == Zstr('\\'); }); + + return concatenateFtpFolderPathPhrase(loginTmp, sanitizeRootRelativePath(relPath)); +} + + +//syntax: ftp://[<user>[:<password>]@]<server>[:port]/<relative-path>[|option_name=value] +// +// e.g. ftp://user001:secretpassword@private.example.com:222/mydirectory/ +// ftp://user001@private.example.com/mydirectory|pass64=c2VjcmV0cGFzc3dvcmQ +FtpPathInfo fff::getResolvedFtpPath(const Zstring& folderPathPhrase) //noexcept +{ + Zstring pathPhrase = expandMacros(folderPathPhrase); //expand before trimming! + trim(pathPhrase); + + if (startsWithAsciiNoCase(pathPhrase, ftpPrefix)) + pathPhrase = pathPhrase.c_str() + strLength(ftpPrefix); + trim(pathPhrase, true, false, [](Zchar c) { return c == Zstr('/') || c == Zstr('\\'); }); + + const Zstring credentials = beforeFirst(pathPhrase, Zstr('@'), IF_MISSING_RETURN_NONE); + const Zstring fullPathOpt = afterFirst(pathPhrase, Zstr('@'), IF_MISSING_RETURN_ALL); + + FtpLoginInfo login; + login.username = decodeFtpUsername(beforeFirst(credentials, Zstr(':'), IF_MISSING_RETURN_ALL)); //support standard FTP syntax, even though ":" + login.password = afterFirst(credentials, Zstr(':'), IF_MISSING_RETURN_NONE); //is not used by our concatenateSftpFolderPathPhrase()! + + const Zstring fullPath = beforeFirst(fullPathOpt, Zstr('|'), IF_MISSING_RETURN_ALL); + const Zstring options = afterFirst(fullPathOpt, Zstr('|'), IF_MISSING_RETURN_NONE); + + auto it = std::find_if(fullPath.begin(), fullPath.end(), [](Zchar c) { return c == '/' || c == '\\'; }); + const Zstring serverPort(fullPath.begin(), it); + const AfsPath serverRelPath = sanitizeRootRelativePath({ it, fullPath.end() }); + + login.server = beforeLast(serverPort, Zstr(':'), IF_MISSING_RETURN_ALL); + const Zstring port = afterLast(serverPort, Zstr(':'), IF_MISSING_RETURN_NONE); + login.port = stringTo<int>(port); //0 if empty + + if (!options.empty()) + { + for (const Zstring& optPhrase : split(options, Zstr("|"), SplitType::SKIP_EMPTY)) + if (startsWith(optPhrase, Zstr("timeout="))) + login.timeoutSec = stringTo<int>(afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE)); + else if (optPhrase == Zstr("ssl")) + login.useSsl = true; + else if (startsWith(optPhrase, Zstr("pass64="))) + login.password = decodePasswordBase64(afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE)); + else + assert(false); + } //fix "-Wdangling-else" + return { login, serverRelPath }; +} + + +bool fff::acceptsItemPathPhraseFtp(const Zstring& itemPathPhrase) //noexcept +{ + Zstring path = expandMacros(itemPathPhrase); //expand before trimming! + trim(path); + return startsWithAsciiNoCase(path, ftpPrefix); //check for explicit FTP path +} + + +AbstractPath fff::createItemPathFtp(const Zstring& itemPathPhrase) //noexcept +{ + const FtpPathInfo pi = getResolvedFtpPath(itemPathPhrase); //noexcept + return AbstractPath(makeSharedRef<FtpFileSystem>(pi.login), pi.afsPath); +} diff --git a/FreeFileSync/Source/fs/ftp.h b/FreeFileSync/Source/fs/ftp.h new file mode 100644 index 00000000..f73d523f --- /dev/null +++ b/FreeFileSync/Source/fs/ftp.h @@ -0,0 +1,43 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef FTP_H_745895742383425326568678 +#define FTP_H_745895742383425326568678 + +#include "abstract.h" + + +namespace fff +{ +bool acceptsItemPathPhraseFtp(const Zstring& itemPathPhrase); //noexcept +AbstractPath createItemPathFtp(const Zstring& itemPathPhrase); //noexcept + +//------------------------------------------------------- + +struct FtpLoginInfo +{ + Zstring server; + int port = 0; // > 0 if set + Zstring username; + Zstring password; + bool useSsl = false; + + //other settings not specific to FTP session: + int timeoutSec = 15; +}; + +struct FtpPathInfo +{ + FtpLoginInfo login; + AfsPath afsPath; //server-relative path +}; +FtpPathInfo getResolvedFtpPath(const Zstring& folderPathPhrase); //noexcept + +//expects (potentially messy) user input: +Zstring condenseToFtpFolderPathPhrase(const FtpLoginInfo& login, const Zstring& relPath); //noexcept +} + +#endif //FTP_H_745895742383425326568678 diff --git a/FreeFileSync/Source/fs/ftp_common.h b/FreeFileSync/Source/fs/ftp_common.h new file mode 100644 index 00000000..edd3da54 --- /dev/null +++ b/FreeFileSync/Source/fs/ftp_common.h @@ -0,0 +1,68 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef FTP_COMMON_H_92889457091324321454 +#define FTP_COMMON_H_92889457091324321454 + +#include <zen/base64.h> +#include "abstract.h" + + +namespace fff +{ +inline +Zstring encodePasswordBase64(const Zstring& pass) +{ + using namespace zen; + return utfTo<Zstring>(stringEncodeBase64(utfTo<std::string>(pass))); //nothrow +} + + +inline +Zstring decodePasswordBase64(const Zstring& pass) +{ + using namespace zen; + return utfTo<Zstring>(stringDecodeBase64(utfTo<std::string>(pass))); //nothrow +} + + +//according to the SFTP path syntax, the user name must not contain raw @ and : +//-> we don't need a full urlencode! +inline +Zstring encodeFtpUsername(Zstring name) +{ + using namespace zen; + replace(name, Zstr("%"), Zstr("%25")); //first! + replace(name, Zstr("@"), Zstr("%40")); + replace(name, Zstr(":"), Zstr("%3A")); + return name; +} + + +inline +Zstring decodeFtpUsername(Zstring name) +{ + using namespace zen; + replace(name, Zstr("%40"), Zstr("@")); + replace(name, Zstr("%3A"), Zstr(":")); + replace(name, Zstr("%3a"), Zstr(":")); + replace(name, Zstr("%25"), Zstr("%")); //last! + return name; +} + + +//(S)FTP path relative to server root using Unix path separators and with leading slash +inline +Zstring getServerRelPath(const AfsPath& afsPath) +{ + if constexpr (FILE_NAME_SEPARATOR != Zstr('/' )) + return Zstr('/') + replaceCpy(afsPath.value, FILE_NAME_SEPARATOR, Zstr('/')); + else + return Zstr('/') + afsPath.value; +} +} + +#endif //FTP_COMMON_H_92889457091324321454 diff --git a/FreeFileSync/Source/fs/gdrive.cpp b/FreeFileSync/Source/fs/gdrive.cpp new file mode 100644 index 00000000..f8044a57 --- /dev/null +++ b/FreeFileSync/Source/fs/gdrive.cpp @@ -0,0 +1,3163 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "gdrive.h" +#include <variant> +#include <unordered_map> +#include <unordered_set> +#include <zen/base64.h> +#include <zen/basic_math.h> +#include <zen/file_traverser.h> +#include <zen/shell_execute.h> +#include <zen/http.h> +#include <zen/zlib_wrap.h> +#include <zen/crc.h> +#include <zen/json.h> +#include <zen/time.h> +#include <zen/file_access.h> +#include <zen/guid.h> +#include <zen/socket.h> +#include <zen/file_io.h> +#include "abstract_impl.h" +#include "libcurl/curl_wrap.h" //DON'T include <curl/curl.h> directly! +#include "init_curl_libssh2.h" +#include "../base/resolve_path.h" + +using namespace zen; +using namespace fff; +using AFS = AbstractFileSystem; + + +namespace fff +{ +bool operator<(const GdrivePath& lhs, const GdrivePath& rhs) +{ + const int rv = compareAsciiNoCase(lhs.userEmail, rhs.userEmail); + if (rv != 0) + return rv < 0; + + //mirror GoogleFileState file path matching + return compareNativePath(lhs.itemPath.value, rhs.itemPath.value) < 0; +} + + +Global<PathAccessLocker<GdrivePath>> globalGdrivePathAccessLocker(std::make_unique<PathAccessLocker<GdrivePath>>()); +template <> std::shared_ptr<PathAccessLocker<GdrivePath>> PathAccessLocker<GdrivePath>::getGlobalInstance() { return globalGdrivePathAccessLocker.get(); } +using PathAccessLock = PathAccessLocker<GdrivePath>::Lock; //throw SysError +} + + +namespace +{ +//Google Drive REST API Overview: https://developers.google.com/drive/api/v3/about-sdk +//Google Drive REST API Reference: https://developers.google.com/drive/api/v3/reference + +//Google Drive credentials https://console.developers.google.com/apis/credentials?project=freefilesync-217608 + const char* GOOGLE_DRIVE_CLIENT_ID = ""; // => replace with live credentials + const char* GOOGLE_DRIVE_CLIENT_SECRET = ""; // +const Zchar* GOOGLE_REST_API_SERVER = Zstr("www.googleapis.com"); + +const std::chrono::seconds HTTP_SESSION_ACCESS_TIME_OUT(15); +const std::chrono::seconds HTTP_SESSION_MAX_IDLE_TIME (20); +const std::chrono::seconds HTTP_SESSION_CLEANUP_INTERVAL(4); +const std::chrono::seconds GOOGLE_DRIVE_SYNC_INTERVAL (5); + +const int GDRIVE_STREAM_BUFFER_SIZE = 512 * 1024; //unit: [byte] + +const Zchar googleDrivePrefix[] = Zstr("gdrive:"); +const char googleFolderMimeType[] = "application/vnd.google-apps.folder"; + +const char DB_FORMAT_DESCR[] = "FreeFileSync: Google Drive Database"; +const int DB_FORMAT_VER = 1; + + +struct HttpSessionId +{ + /*explicit*/ HttpSessionId(const Zstring& serverName) : + server(serverName) {} + + Zstring server; +}; +bool operator<(const HttpSessionId& lhs, const HttpSessionId& rhs) +{ + //exactly the type of case insensitive comparison we need for server names! + return compareAsciiNoCase(lhs.server, rhs.server) < 0; //https://msdn.microsoft.com/en-us/library/windows/desktop/ms738519#IDNs +} + + +//expects "clean" input data, see condenseToGoogleFolderPathPhrase() +Zstring concatenateGoogleFolderPathPhrase(const GdrivePath& gdrivePath) //noexcept +{ + Zstring pathPhrase = Zstring(googleDrivePrefix) + FILE_NAME_SEPARATOR + gdrivePath.userEmail; + if (!gdrivePath.itemPath.value.empty()) + pathPhrase += FILE_NAME_SEPARATOR + gdrivePath.itemPath.value; + return pathPhrase; +} + + +//e.g.: gdrive:/zenju@gmx.net/folder/file.txt +std::wstring getGoogleDisplayPath(const GdrivePath& gdrivePath) +{ + return utfTo<std::wstring>(concatenateGoogleFolderPathPhrase(gdrivePath)); //noexcept +} + + +std::wstring formatGoogleErrorRaw(const std::string& serverResponse) +{ + /* e.g.: { "error": { "errors": [{ "domain": "global", + "reason": "invalidSharingRequest", + "message": "Bad Request. User message: \"ACL change not allowed.\"" }], + "code": 400, + "message": "Bad Request" }} + + or: { "error": "invalid_client", + "error_description": "Unauthorized" } + + or merely: { "error": "invalid_token" } */ + try + { + const JsonValue jresponse = parseJson(serverResponse); //throw JsonParsingError + + if (const JsonValue* error = getChildFromJsonObject(jresponse, "error")) + { + if (error->type == JsonValue::Type::string) + return utfTo<std::wstring>(error->primVal); + //the inner message is generally more descriptive! + else if (const JsonValue* errors = getChildFromJsonObject(*error, "errors")) + if (errors->type == JsonValue::Type::array && !errors->arrayVal.empty()) + if (const JsonValue* message = getChildFromJsonObject(*errors->arrayVal[0], "message")) + if (message->type == JsonValue::Type::string) + return utfTo<std::wstring>(message->primVal); + } + } + catch (JsonParsingError&) {} //not JSON? + + assert(false); + return utfTo<std::wstring>(serverResponse); +} + + +std::wstring tryFormatHttpErrorCode(int ec) //https://en.wikipedia.org/wiki/List_of_HTTP_status_codes +{ + if (ec == 300) return L"Multiple Choices."; + if (ec == 301) return L"Moved Permanently."; + if (ec == 302) return L"Moved temporarily."; + if (ec == 303) return L"See Other"; + if (ec == 304) return L"Not Modified."; + if (ec == 305) return L"Use Proxy."; + if (ec == 306) return L"Switch Proxy."; + if (ec == 307) return L"Temporary Redirect."; + if (ec == 308) return L"Permanent Redirect."; + + if (ec == 400) return L"Bad Request."; + if (ec == 401) return L"Unauthorized."; + if (ec == 402) return L"Payment Required."; + if (ec == 403) return L"Forbidden."; + if (ec == 404) return L"Not Found."; + if (ec == 405) return L"Method Not Allowed."; + if (ec == 406) return L"Not Acceptable."; + if (ec == 407) return L"Proxy Authentication Required."; + if (ec == 408) return L"Request Timeout."; + if (ec == 409) return L"Conflict."; + if (ec == 410) return L"Gone."; + if (ec == 411) return L"Length Required."; + if (ec == 412) return L"Precondition Failed."; + if (ec == 413) return L"Payload Too Large."; + if (ec == 414) return L"URI Too Long."; + if (ec == 415) return L"Unsupported Media Type."; + if (ec == 416) return L"Range Not Satisfiable."; + if (ec == 417) return L"Expectation Failed."; + if (ec == 418) return L"I\'m a teapot."; + if (ec == 421) return L"Misdirected Request."; + if (ec == 422) return L"Unprocessable Entity."; + if (ec == 423) return L"Locked."; + if (ec == 424) return L"Failed Dependency."; + if (ec == 426) return L"Upgrade Required."; + if (ec == 428) return L"Precondition Required."; + if (ec == 429) return L"Too Many Requests."; + if (ec == 431) return L"Request Header Fields Too Large."; + if (ec == 451) return L"Unavailable For Legal Reasons."; + + if (ec == 500) return L"Internal Server Error."; + if (ec == 501) return L"Not Implemented."; + if (ec == 502) return L"Bad Gateway."; + if (ec == 503) return L"Service Unavailable."; + if (ec == 504) return L"Gateway Timeout."; + if (ec == 505) return L"HTTP Version Not Supported."; + if (ec == 506) return L"Variant Also Negotiates."; + if (ec == 507) return L"Insufficient Storage."; + if (ec == 508) return L"Loop Detected."; + if (ec == 510) return L"Not Extended."; + if (ec == 511) return L"Network Authentication Required."; + return L""; +} +//---------------------------------------------------------------------------------------------------------------- + +Global<UniSessionCounter> httpSessionCount(createUniSessionCounter()); + + +class HttpSession +{ +public: + HttpSession(const HttpSessionId& sessionId, const Zstring& caCertFilePath) : //throw FileError + sessionId_(sessionId), + caCertFilePath_(utfTo<std::string>(caCertFilePath)) + { + try + { + libsshCurlUnifiedInitCookie_ = getLibsshCurlUnifiedInitCookie(httpSessionCount); //throw SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + } + + ~HttpSession() + { + if (easyHandle_) + ::curl_easy_cleanup(easyHandle_); + } + + struct Option + { + template <class T> + Option(CURLoption o, T val) : option(o), value(static_cast<uint64_t>(val)) { static_assert(sizeof(val) <= sizeof(value)); } + + template <class T> + Option(CURLoption o, T* val) : option(o), value(reinterpret_cast<uint64_t>(val)) { static_assert(sizeof(val) <= sizeof(value)); } + + CURLoption option = CURLOPT_LASTENTRY; + uint64_t value = 0; + }; + + struct HttpResult + { + int statusCode = 0; + //std::string contentType; + }; + HttpResult perform(const std::string& serverRelPath, + const std::vector<std::string>& extraHeaders, const std::vector<Option>& extraOptions, //throw FileError + const std::function<void (const void* buffer, size_t bytesToWrite)>& writeResponse /*throw X*/, //optional + const std::function<size_t( void* buffer, size_t bytesToRead )>& readRequest /*throw X*/) // + { + if (!easyHandle_) + { + easyHandle_ = ::curl_easy_init(); + if (!easyHandle_) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"curl_easy_init", formatCurlErrorRaw(CURLE_OUT_OF_MEMORY), std::wstring())); + } + else + ::curl_easy_reset(easyHandle_); + + + std::vector<Option> options; + + curlErrorBuf_[0] = '\0'; + options.emplace_back(CURLOPT_ERRORBUFFER, curlErrorBuf_); + + options.emplace_back(CURLOPT_USERAGENT, "FreeFileSync"); //default value; may be overwritten by caller + + //lifetime: keep alive until after curl_easy_setopt() below + std::string curlPath = "https://" + utfTo<std::string>(sessionId_.server) + serverRelPath; + options.emplace_back(CURLOPT_URL, curlPath.c_str()); + + options.emplace_back(CURLOPT_NOSIGNAL, 1L); //thread-safety: https://curl.haxx.se/libcurl/c/threadsafe.html + + options.emplace_back(CURLOPT_CONNECTTIMEOUT, std::chrono::seconds(HTTP_SESSION_ACCESS_TIME_OUT).count()); + + //CURLOPT_TIMEOUT: "Since this puts a hard limit for how long time a request is allowed to take, it has limited use in dynamic use cases with varying transfer times." + options.emplace_back(CURLOPT_LOW_SPEED_TIME, std::chrono::seconds(HTTP_SESSION_ACCESS_TIME_OUT).count()); + options.emplace_back(CURLOPT_LOW_SPEED_LIMIT, 1L); //[bytes], can't use "0" which means "inactive", so use some low number + + + //libcurl forwards this char-string to OpenSSL as is, which (thank god) accepts UTF8 + options.emplace_back(CURLOPT_CAINFO, caCertFilePath_.c_str()); //hopefully latest version from https://curl.haxx.se/docs/caextract.html + //CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST are already active by default + + //--------------------------------------------------- + std::exception_ptr userCallbackException; + + auto onBytesReceived = [&](const void* buffer, size_t len) + { + try + { + writeResponse(buffer, len); //throw X + return len; + } + catch (...) + { + userCallbackException = std::current_exception(); + return len + 1; //signal error condition => CURLE_WRITE_ERROR + } + }; + using ReadCbType = decltype(onBytesReceived); + using ReadCbWrapperType = size_t (*)(const void* buffer, size_t size, size_t nitems, void* callbackData); //needed for cdecl function pointer cast + ReadCbWrapperType onBytesReceivedWrapper = [](const void* buffer, size_t size, size_t nitems, void* callbackData) + { + auto cb = static_cast<ReadCbType*>(callbackData); //free this poor little C-API from its shackles and redirect to a proper lambda + return (*cb)(buffer, size * nitems); + }; + //--------------------------------------------------- + auto getBytesToSend = [&](void* buffer, size_t len) -> size_t + { + try + { + //libcurl calls back until 0 bytes are returned (Posix read() semantics), or, + //if CURLOPT_INFILESIZE_LARGE was set, after exactly this amount of bytes + const size_t bytesRead = readRequest(buffer, len);//throw X; return "bytesToRead" bytes unless end of stream! + return bytesRead; + } + catch (...) + { + userCallbackException = std::current_exception(); + return CURL_READFUNC_ABORT; //signal error condition => CURLE_ABORTED_BY_CALLBACK + } + }; + using WriteCbType = decltype(getBytesToSend); + using WriteCbWrapperType = size_t (*)(void* buffer, size_t size, size_t nitems, void* callbackData); + WriteCbWrapperType getBytesToSendWrapper = [](void* buffer, size_t size, size_t nitems, void* callbackData) + { + auto cb = static_cast<WriteCbType*>(callbackData); //free this poor little C-API from its shackles and redirect to a proper lambda + return (*cb)(buffer, size * nitems); + }; + //--------------------------------------------------- + if (writeResponse) + { + options.emplace_back(CURLOPT_WRITEDATA, &onBytesReceived); + options.emplace_back(CURLOPT_WRITEFUNCTION, onBytesReceivedWrapper); + } + if (readRequest) + { + if (std::all_of(extraOptions.begin(), extraOptions.end(), [](const Option& o) { return o.option != CURLOPT_POST; })) + options.emplace_back(CURLOPT_UPLOAD, 1L); //issues HTTP PUT + options.emplace_back(CURLOPT_READDATA, &getBytesToSend); + options.emplace_back(CURLOPT_READFUNCTION, getBytesToSendWrapper); + } + + if (std::any_of(extraOptions.begin(), extraOptions.end(), [](const Option& o) { return o.option == CURLOPT_WRITEFUNCTION || o.option == CURLOPT_READFUNCTION; })) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); //Option already used here! + + if (readRequest && std::any_of(extraOptions.begin(), extraOptions.end(), [](const Option& o) { return o.option == CURLOPT_POSTFIELDS; })) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); //Contradicting options: CURLOPT_READFUNCTION, CURLOPT_POSTFIELDS + + //--------------------------------------------------- + struct curl_slist* headers = nullptr; //"libcurl will not copy the entire list so you must keep it!" + ZEN_ON_SCOPE_EXIT(::curl_slist_free_all(headers)); + for (const std::string& headerLine : extraHeaders) + headers = ::curl_slist_append(headers, headerLine.c_str()); + + if (headers) + options.emplace_back(CURLOPT_HTTPHEADER, headers); + //--------------------------------------------------- + + append(options, extraOptions); + + for (const Option& opt : options) + { + const CURLcode rc = ::curl_easy_setopt(easyHandle_, opt.option, opt.value); + if (rc != CURLE_OK) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"curl_easy_setopt " + numberTo<std::wstring>(opt.option), + formatCurlErrorRaw(rc), utfTo<std::wstring>(::curl_easy_strerror(rc)))); + } + + //======================================================================================================= + const CURLcode rcPerf = ::curl_easy_perform(easyHandle_); + //WTF: curl_easy_perform() considers FTP response codes 4XX, 5XX as failure, but for HTTP response codes 4XX are considered success!! CONSISTENCY, people!!! + //=> at least libcurl is aware: CURLOPT_FAILONERROR: "request failure on HTTP response >= 400"; default: "0, do not fail on error" + //=> Curiously Google also screws up in their REST API design and returns HTTP 4XX status for domain-level errors! + //=> let caller handle HTTP status to work around this mess! + + if (userCallbackException) + std::rethrow_exception(userCallbackException); //throw X + //======================================================================================================= + + try + { + long httpStatusCode = 0; //optional + { + const CURLcode rc = ::curl_easy_getinfo(easyHandle_, CURLINFO_RESPONSE_CODE, &httpStatusCode); + if (rc != CURLE_OK) + throw SysError(formatSystemError(L"curl_easy_getinfo: CURLINFO_RESPONSE_CODE", formatCurlErrorRaw(rc), utfTo<std::wstring>(::curl_easy_strerror(rc)))); + } + //char* contentType = nullptr; //optional; owned by libcurl + //{ + // const CURLcode rc = ::curl_easy_getinfo(easyHandle_, CURLINFO_CONTENT_TYPE, &contentType); + // if (rc != CURLE_OK) + // throw SysError(formatSystemError(L"curl_easy_getinfo: CURLINFO_CONTENT_TYPE", formatCurlErrorRaw(rc), utfTo<std::wstring>(::curl_easy_strerror(rc)))); + //} + + if (rcPerf != CURLE_OK) + throw SysError(formatLastCurlError(L"curl_easy_perform", rcPerf, httpStatusCode)); + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + return { static_cast<int>(httpStatusCode) /*, contentType ? contentType : ""*/ }; + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + } + + //------------------------------------------------------------------------------------------------------------ + + bool isHealthy() const + { + return numeric::dist(std::chrono::steady_clock::now(), lastSuccessfulUseTime_) <= HTTP_SESSION_MAX_IDLE_TIME; + } + + const HttpSessionId& getSessionId() const { return sessionId_; } + +private: + HttpSession (const HttpSession&) = delete; + HttpSession& operator=(const HttpSession&) = delete; + + std::wstring formatLastCurlError(const std::wstring& functionName, CURLcode ec, int httpStatusCode) const + { + std::wstring errorMsg; + + if (curlErrorBuf_[0] != 0) + errorMsg = trimCpy(utfTo<std::wstring>(curlErrorBuf_)); + + const std::wstring descr = tryFormatHttpErrorCode(httpStatusCode); + if (!descr.empty()) + errorMsg += (errorMsg.empty() ? L"" : L"\n") + numberTo<std::wstring>(httpStatusCode) + L": " + descr; +#if 0 + //utfTo<std::wstring>(::curl_easy_strerror(ec)) is uninteresting + //use CURLINFO_OS_ERRNO ?? https://curl.haxx.se/libcurl/c/CURLINFO_OS_ERRNO.html + long nativeErrorCode = 0; + if (::curl_easy_getinfo(easyHandle_, CURLINFO_OS_ERRNO, &nativeErrorCode) == CURLE_OK) + if (nativeErrorCode != 0) + errorMsg += (errorMsg.empty() ? L"" : L"\n") + std::wstring(L"Native error code: ") + numberTo<std::wstring>(nativeErrorCode); +#endif + return formatSystemError(functionName, formatCurlErrorRaw(ec), errorMsg); + } + + const HttpSessionId sessionId_; + const std::string caCertFilePath_; + CURL* easyHandle_ = nullptr; + char curlErrorBuf_[CURL_ERROR_SIZE] = {}; + + std::chrono::steady_clock::time_point lastSuccessfulUseTime_; + std::shared_ptr<UniCounterCookie> libsshCurlUnifiedInitCookie_; +}; + +//---------------------------------------------------------------------------------------------------------------- +//---------------------------------------------------------------------------------------------------------------- + +class HttpSessionManager //reuse (healthy) HTTP sessions globally +{ +public: + HttpSessionManager(const Zstring& caCertFilePath) : caCertFilePath_(caCertFilePath), + sessionCleaner_([this] + { + setCurrentThreadName("Session Cleaner[HTTP]"); + runGlobalSessionCleanUp(); //throw ThreadInterruption + }) {} + + ~HttpSessionManager() + { + sessionCleaner_.interrupt(); + sessionCleaner_.join(); + } + + using IdleHttpSessions = std::vector<std::unique_ptr<HttpSession>>; + + void access(const HttpSessionId& login, const std::function<void(HttpSession& session)>& useHttpSession /*throw X*/) //throw FileError, X + { + Protected<HttpSessionManager::IdleHttpSessions>& sessionStore = getSessionStore(login); + + std::unique_ptr<HttpSession> httpSession; + + sessionStore.access([&](HttpSessionManager::IdleHttpSessions& sessions) + { + //assume "isHealthy()" to avoid hitting server connection limits: (clean up of !isHealthy() after use, idle sessions via worker thread) + if (!sessions.empty()) + { + httpSession = std::move(sessions.back ()); + /**/ sessions.pop_back(); + } + }); + + //create new HTTP session outside the lock: 1. don't block other threads 2. non-atomic regarding "sessionStore"! => one session too many is not a problem! + if (!httpSession) + httpSession = std::make_unique<HttpSession>(login, caCertFilePath_); //throw FileError + + ZEN_ON_SCOPE_EXIT( + if (httpSession->isHealthy()) //thread that created the "!isHealthy()" session is responsible for clean up (avoid hitting server connection limits!) + sessionStore.access([&](HttpSessionManager::IdleHttpSessions& sessions) { sessions.push_back(std::move(httpSession)); }); ); + + useHttpSession(*httpSession); //throw X + } + +private: + HttpSessionManager (const HttpSessionManager&) = delete; + HttpSessionManager& operator=(const HttpSessionManager&) = delete; + + Protected<IdleHttpSessions>& getSessionStore(const HttpSessionId& login) + { + //single global session store per login; life-time bound to globalInstance => never remove a sessionStore!!! + Protected<IdleHttpSessions>* store = nullptr; + + globalSessionStore_.access([&](GlobalHttpSessions& sessionsById) + { + store = &sessionsById[login]; //get or create + }); + static_assert(std::is_same_v<GlobalHttpSessions, std::map<HttpSessionId, Protected<IdleHttpSessions>>>, "require std::map so that the pointers we return remain stable"); + + return *store; + } + + //run a dedicated clean-up thread => it's unclear when the server let's a connection time out, so we do it preemptively + //context of worker thread: + void runGlobalSessionCleanUp() //throw ThreadInterruption + { + std::chrono::steady_clock::time_point lastCleanupTime; + for (;;) + { + const auto now = std::chrono::steady_clock::now(); + + if (now < lastCleanupTime + HTTP_SESSION_CLEANUP_INTERVAL) + interruptibleSleep(lastCleanupTime + HTTP_SESSION_CLEANUP_INTERVAL - now); //throw ThreadInterruption + + lastCleanupTime = std::chrono::steady_clock::now(); + + std::vector<Protected<IdleHttpSessions>*> sessionStores; //pointers remain stable, thanks to std::map<> + + globalSessionStore_.access([&](GlobalHttpSessions& sessionsById) + { + for (auto& [sessionId, idleSession] : sessionsById) + sessionStores.push_back(&idleSession); + }); + + for (Protected<IdleHttpSessions>* sessionStore : sessionStores) + for (bool done = false; !done;) + sessionStore->access([&](IdleHttpSessions& sessions) + { + for (std::unique_ptr<HttpSession>& sshSession : sessions) + if (!sshSession->isHealthy()) //!isHealthy() sessions are destroyed after use => in this context this means they have been idle for too long + { + sshSession.swap(sessions.back()); + /**/ sessions.pop_back(); //run ~HttpSession *inside* the lock! => avoid hitting server limits! + std::this_thread::yield(); + return; //don't hold lock for too long: delete only one session at a time, then yield... + } + done = true; + }); + } + } + + using GlobalHttpSessions = std::map<HttpSessionId, Protected<IdleHttpSessions>>; + + Protected<GlobalHttpSessions> globalSessionStore_; + const Zstring caCertFilePath_; + InterruptibleThread sessionCleaner_; +}; + +//-------------------------------------------------------------------------------------- +UniInitializer startupInitHttp(*httpSessionCount.get()); //static ordering: place *before* HttpSessionManager instance! + +Global<HttpSessionManager> httpSessionManager; +//-------------------------------------------------------------------------------------- + + +//=========================================================================================================================== + +//try to get a grip on this crazy REST API: - parameters are passed via query string, header, or body, using GET, POST, PUT, PATCH, DELETE, ... it's a dice roll +HttpSession::HttpResult googleHttpsRequest(const std::string& serverRelPath, //throw FileError + const std::vector<std::string>& extraHeaders, + const std::vector<HttpSession::Option>& extraOptions, + const std::function<void (const void* buffer, size_t bytesToWrite)>& writeResponse /*throw X*/, //optional + const std::function<size_t( void* buffer, size_t bytesToRead )>& readRequest /*throw X*/) //optional; returning 0 signals EOF +{ + const std::shared_ptr<HttpSessionManager> mgr = httpSessionManager.get(); + if (!mgr) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(GOOGLE_REST_API_SERVER)), L"Function call not allowed during process init/shutdown."); + + HttpSession::HttpResult httpResult; + + mgr->access(HttpSessionId(GOOGLE_REST_API_SERVER), [&](HttpSession& session) //throw FileError + { + std::vector<HttpSession::Option> options = + { + //https://developers.google.com/drive/api/v3/performance + //"In order to receive a gzip-encoded response you must do two things: Set an Accept-Encoding header, and modify your user agent to contain the string gzip." + { CURLOPT_ACCEPT_ENCODING, "gzip" }, + { CURLOPT_USERAGENT, "FreeFileSync (gzip)" }, + }; + append(options, extraOptions); + + httpResult = session.perform(serverRelPath, extraHeaders, options, writeResponse, readRequest); //throw FileError + }); + return httpResult; +} + +//======================================================================================================== + +struct GoogleUserInfo +{ + std::wstring displayName; + Zstring email; +}; +GoogleUserInfo getUserInfo(const std::string& accessToken) //throw FileError +{ + //https://developers.google.com/drive/api/v3/reference/about + const std::string queryParams = xWwwFormUrlEncode( + { + { "fields", "user/displayName,user/emailAddress" }, + }); + std::string response; + googleHttpsRequest("/drive/v3/about?" + queryParams, { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + if (const JsonValue* user = getChildFromJsonObject(jresponse, "user")) + { + const std::optional<std::string> displayName = getPrimitiveFromJsonObject(*user, "displayName"); + const std::optional<std::string> email = getPrimitiveFromJsonObject(*user, "emailAddress"); + if (displayName && email) + return { utfTo<std::wstring>(*displayName), utfTo<Zstring>(*email) }; + } + + throw FileError(replaceCpy(_("Failed to get information about server %x."), L"%x", fmtPath(Zstr("Google Drive"))), formatGoogleErrorRaw(response)); +} + + +const char* htmlMessageTemplate = R""(<!doctype html> +<html lang="en"> + <head> + <meta charset="utf-8"> + <title>TITLE_PLACEHOLDER</title> + <style type="text/css"> + * { + font-family: "Helvetica Neue", "Segoe UI", Segoe, Helvetica, Arial, "Lucida Grande", sans-serif; + text-align: center; + background-color: #eee; } + h1 { + font-size: 45px; + font-weight: 300; + margin: 80px 0 20px 0; } + .descr { + font-size: 21px; + font-weight: 200; } + </style> + </head> + <body> + <h1><img src="https://freefilesync.org/images/FreeFileSync.png" style="vertical-align:middle; height: 50px;" alt=""> TITLE_PLACEHOLDER</h1> + <div class="descr">MESSAGE_PLACEHOLDER</div> + </body> +</html> +)""; + +struct GoogleAuthCode +{ + std::string code; + std::string redirectUrl; + std::string codeChallenge; +}; + +struct GoogleAccessToken +{ + std::string value; + time_t validUntil; //remaining lifetime of the access token +}; + +struct GoogleAccessInfo +{ + GoogleAccessToken accessToken; + std::string refreshToken; + GoogleUserInfo userInfo; +}; + +GoogleAccessInfo googleDriveExchangeAuthCode(const GoogleAuthCode& authCode) //throw FileError +{ + //https://developers.google.com/identity/protocols/OAuth2InstalledApp#exchange-authorization-code + const std::string postBuf = xWwwFormUrlEncode( + { + { "code", authCode.code }, + { "client_id", GOOGLE_DRIVE_CLIENT_ID }, + { "client_secret", GOOGLE_DRIVE_CLIENT_SECRET }, + { "redirect_uri", authCode.redirectUrl }, + { "grant_type", "authorization_code" }, + { "code_verifier", authCode.codeChallenge }, + }); + std::string response; + googleHttpsRequest("/oauth2/v4/token", {} /*extraHeaders*/, { { CURLOPT_POSTFIELDS, postBuf.c_str() } }, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> accessToken = getPrimitiveFromJsonObject(jresponse, "access_token"); + const std::optional<std::string> refreshToken = getPrimitiveFromJsonObject(jresponse, "refresh_token"); + const std::optional<std::string> expiresIn = getPrimitiveFromJsonObject(jresponse, "expires_in"); //e.g. 3600 seconds + if (!accessToken || !refreshToken || !expiresIn) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", L"Google Drive"), formatGoogleErrorRaw(response)); + + const GoogleUserInfo userInfo = getUserInfo(*accessToken); //throw FileError + + return { { *accessToken, std::time(nullptr) + stringTo<time_t>(*expiresIn) }, *refreshToken, userInfo }; +} + + +GoogleAccessInfo authorizeAccessToGoogleDrive(const Zstring& googleLoginHint, const std::function<void()>& updateGui /*throw X*/) //throw FileError, X +{ + try //spin up a web server to wait for the HTTP GET after Google authentication + { + ::addrinfo hints = {}; + hints.ai_family = AF_INET; //make sure our server is reached by IPv4 127.0.0.1, not IPv6 [::1] + hints.ai_socktype = SOCK_STREAM; //we *do* care about this one! + hints.ai_flags = AI_PASSIVE; //the returned socket addresses will be suitable for bind(2)ing a socket that will accept(2) connections. + hints.ai_flags |= AI_ADDRCONFIG; //no such issue on Linux: https://bugs.chromium.org/p/chromium/issues/detail?id=5234 + ::addrinfo* servinfo = nullptr; + ZEN_ON_SCOPE_EXIT(if (servinfo) ::freeaddrinfo(servinfo)); + + //ServiceName == "0" => open the next best free port + const int rcGai = ::getaddrinfo(nullptr, //_In_opt_ PCSTR pNodeName, + "0", //_In_opt_ PCSTR pServiceName, + &hints, //_In_opt_ const ADDRINFOA* pHints, + &servinfo); //_Outptr_ PADDRINFOA* ppResult + if (rcGai != 0) + throw SysError(formatSystemError(L"getaddrinfo", replaceCpy(_("Error Code %x"), L"%x", numberTo<std::wstring>(rcGai)), utfTo<std::wstring>(::gai_strerror(rcGai)))); + if (!servinfo) + throw SysError(L"getaddrinfo: empty server info"); + + auto getBoundSocket = [&](const auto& /*::addrinfo*/ ai) + { + SocketType testSocket = ::socket(ai.ai_family, ai.ai_socktype, ai.ai_protocol); + if (testSocket == invalidSocket) + THROW_LAST_SYS_ERROR_WSA(L"socket"); + ZEN_ON_SCOPE_FAIL(closeSocket(testSocket)); + + if (::bind(testSocket, ai.ai_addr, static_cast<int>(ai.ai_addrlen)) != 0) + THROW_LAST_SYS_ERROR_WSA(L"bind"); + + return testSocket; + }; + + SocketType socket = invalidSocket; + + std::optional<SysError> firstError; + for (const auto* /*::addrinfo*/ si = servinfo; si; si = si->ai_next) + try + { + socket = getBoundSocket(*si); //throw SysError; pass ownership + break; + } + catch (const SysError& e) { if (!firstError) firstError = e; } + + if (socket == invalidSocket) + throw* firstError; //list was not empty, so there must have been an error! + + ZEN_ON_SCOPE_EXIT(closeSocket(socket)); + + + sockaddr_storage addr = {}; //"sufficiently large to store address information for IPv4 or IPv6" => sockaddr_in and sockaddr_in6 + socklen_t addrLen = sizeof(addr); + if (::getsockname(socket, reinterpret_cast<sockaddr*>(&addr), &addrLen) != 0) + THROW_LAST_SYS_ERROR_WSA(L"getsockname"); + + if (addr.ss_family != AF_INET && + addr.ss_family != AF_INET6) + throw SysError(L"getsockname: unknown protocol family (" + numberTo<std::wstring>(addr.ss_family) + L")"); + +const int port = ntohs(reinterpret_cast<const sockaddr_in&>(addr).sin_port); +//the socket is not bound to a specific local IP => inet_ntoa(reinterpret_cast<const sockaddr_in&>(addr).sin_addr) == "0.0.0.0" +const std::string redirectUrl = "http://127.0.0.1:" + numberTo<std::string>(port); + +if (::listen(socket, SOMAXCONN) != 0) + THROW_LAST_SYS_ERROR_WSA(L"listen"); + + + //"A code_verifier is a high-entropy cryptographic random string using the unreserved characters:" + //[A-Z] / [a-z] / [0-9] / "-" / "." / "_" / "~", with a minimum length of 43 characters and a maximum length of 128 characters. + std::string codeChallenge = stringEncodeBase64(generateGUID() + generateGUID()); + replace(codeChallenge, '+', '-'); // + replace(codeChallenge, '/', '.'); //code_verifier is almost a perfect fit for base64! + replace(codeChallenge, '=', '_'); // + assert(codeChallenge.size() == 44); + + //authenticate Google Drive via browser: https://developers.google.com/identity/protocols/OAuth2InstalledApp#step-2-send-a-request-to-googles-oauth-20-server + const std::string oauthUrl = "https://accounts.google.com/o/oauth2/v2/auth?" + xWwwFormUrlEncode( +{ + { "client_id", GOOGLE_DRIVE_CLIENT_ID }, + { "redirect_uri", redirectUrl }, + { "response_type", "code" }, + { "scope", "https://www.googleapis.com/auth/drive" }, + { "code_challenge", codeChallenge }, + { "code_challenge_method", "plain" }, + { "login_hint", utfTo<std::string>(googleLoginHint) }, +}); +openWithDefaultApplication(utfTo<Zstring>(oauthUrl)); //throw FileError +//[!] no need to map to SysError + +//process incoming HTTP requests +for (;;) +{ +for (;;) //::accept() blocks forever if no client connects (e.g. user just closes the browser window!) => wait for incoming traffic with a time-out via ::select() + { + if (updateGui) updateGui(); //throw X + + fd_set rfd = {}; + FD_ZERO(&rfd); + FD_SET(socket, &rfd); + fd_set* readfds = &rfd; + + struct ::timeval tv = {}; + tv.tv_usec = static_cast<long>(100 /*ms*/) * 1000; + + //WSAPoll broken, even ::poll() on OS X? https://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/ + //perf: no significant difference compared to ::WSAPoll() + const int rc = ::select(socket + 1, readfds, nullptr /*writefds*/, nullptr /*errorfds*/, &tv); + if (rc < 0) + THROW_LAST_SYS_ERROR_WSA(L"select"); + if (rc != 0) + break; + //else: time-out! + } + //potential race! if the connection is gone right after ::select() and before ::accept(), latter will hang + const SocketType clientSocket = ::accept(socket, //SOCKET s, + nullptr, //sockaddr *addr, + nullptr); //int *addrlen + if (clientSocket == invalidSocket) + THROW_LAST_SYS_ERROR_WSA(L"accept"); + + //receive first line of HTTP request + std::string reqLine; + for (;;) + { + const size_t blockSize = 64 * 1024; + reqLine.resize(reqLine.size() + blockSize); + const size_t bytesReceived = tryReadSocket(clientSocket, &*(reqLine.end() - blockSize), blockSize); //throw SysError + reqLine.resize(reqLine.size() - blockSize + bytesReceived); //caveat: unsigned arithmetics + + if (contains(reqLine, "\r\n")) + { + reqLine = beforeFirst(reqLine, "\r\n", IF_MISSING_RETURN_NONE); + break; + } + if (bytesReceived == 0 || reqLine.size() >= 100000 /*bogus line length*/) + break; + } + + //get OAuth2.0 authorization result from Google, either: + std::string code; + std::string error; + + //parse header; e.g.: GET http://127.0.0.1:62054/?code=4/ZgBRsB9k68sFzc1Pz1q0__Kh17QK1oOmetySrGiSliXt6hZtTLUlYzm70uElNTH9vt1OqUMzJVeFfplMsYsn4uI HTTP/1.1 + const std::vector<std::string> statusItems = split(reqLine, ' ', SplitType::ALLOW_EMPTY); //Method SP Request-URI SP HTTP-Version CRLF + + if (statusItems.size() == 3 && statusItems[0] == "GET" && startsWith(statusItems[2], "HTTP/")) + { + for (const auto& [name, value] : xWwwFormUrlDecode(afterFirst(statusItems[1], "?", IF_MISSING_RETURN_NONE))) + if (name == "code") + code = value; + else if (name == "error") + error = value; //e.g. "access_denied" => no more detailed error info available :( + } //"add explicit braces to avoid dangling else [-Wdangling-else]" + + std::optional<std::variant<GoogleAccessInfo, FileError>> authResult; + + //send HTTP response; https://www.w3.org/Protocols/HTTP/1.0/spec.html#Request-Line + std::string httpResponse; + if (code.empty() && error.empty()) //parsing error or unrelated HTTP request + httpResponse = "HTTP/1.0 400 Bad Request" "\r\n" "\r\n" "400 Bad Request\n" + reqLine; + else + { + std::string htmlMsg = htmlMessageTemplate; + try + { + if (!error.empty()) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", L"Google Drive"), + replaceCpy(_("Error Code %x"), L"%x", + L"\"" + utfTo<std::wstring>(error) + L"\"")); + + //do as many login-related tasks as possible while we have the browser as an error output device! + //see AFS::connectNetworkFolder() => errors will be lost after time out in dir_exist_async.h! + authResult = googleDriveExchangeAuthCode({ code, redirectUrl, codeChallenge }); //throw FileError + replace(htmlMsg, "TITLE_PLACEHOLDER", utfTo<std::string>(_("Authentication completed."))); + replace(htmlMsg, "MESSAGE_PLACEHOLDER", utfTo<std::string>(_("You may close this page now and continue with FreeFileSync."))); + } + catch (const FileError& e) + { + authResult = e; + replace(htmlMsg, "TITLE_PLACEHOLDER", utfTo<std::string>(_("Authentication failed."))); + replace(htmlMsg, "MESSAGE_PLACEHOLDER", utfTo<std::string>(e.toString())); + } + httpResponse = "HTTP/1.0 200 OK" "\r\n" + "Content-Type: text/html" "\r\n" + "Content-Length: " + numberTo<std::string>(strLength(htmlMsg)) + "\r\n" + "\r\n" + htmlMsg; + } + + for (size_t bytesToSend = httpResponse.size(); bytesToSend > 0;) + bytesToSend -= tryWriteSocket(clientSocket, &*(httpResponse.end() - bytesToSend), bytesToSend); //throw SysError + + shutdownSocketSend(clientSocket); //throw SysError + //--------------------------------------------------------------- + + if (authResult) + { + if (const FileError* e = std::get_if<FileError>(&*authResult)) + throw *e; + return std::get<GoogleAccessInfo>(*authResult); + } +} +} +catch (const SysError& e) +{ + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", L"Google Drive"), e.toString()); +} +} + + +GoogleAccessToken refreshAccessToGoogleDrive(const std::string& refreshToken, const Zstring& googleUserEmail) //throw FileError +{ + //https://developers.google.com/identity/protocols/OAuth2InstalledApp#offline + const std::string postBuf = xWwwFormUrlEncode( + { + { "refresh_token", refreshToken }, + { "client_id", GOOGLE_DRIVE_CLIENT_ID }, + { "client_secret", GOOGLE_DRIVE_CLIENT_SECRET }, + { "grant_type", "refresh_token" }, + }); + + std::string response; + googleHttpsRequest("/oauth2/v4/token", {} /*extraHeaders*/, { { CURLOPT_POSTFIELDS, postBuf.c_str() } }, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> accessToken = getPrimitiveFromJsonObject(jresponse, "access_token"); + const std::optional<std::string> expiresIn = getPrimitiveFromJsonObject(jresponse, "expires_in"); //e.g. 3600 seconds + if (!accessToken || !expiresIn) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), formatGoogleErrorRaw(response)); + + return { *accessToken, std::time(nullptr) + stringTo<time_t>(*expiresIn) }; +} + + +void revokeAccessToGoogleDrive(const std::string& accessToken, const Zstring& googleUserEmail) //throw FileError +{ + //https://developers.google.com/identity/protocols/OAuth2InstalledApp#tokenrevoke + const std::shared_ptr<HttpSessionManager> mgr = httpSessionManager.get(); + if (!mgr) + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), + L"Function call not allowed during process init/shutdown."); + + HttpSession::HttpResult httpResult; + std::string response; + + mgr->access(HttpSessionId(Zstr("accounts.google.com")), [&](HttpSession& session) //throw FileError + { + httpResult = session.perform("/o/oauth2/revoke?token=" + accessToken, { "Content-Type: application/x-www-form-urlencoded" }, {} /*extraOptions*/, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); //throw FileError + }); + + if (httpResult.statusCode != 200) + throw FileError(replaceCpy(_("Unable to disconnect from %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), formatGoogleErrorRaw(response)); +} + + +uint64_t gdriveGetFreeDiskSpace(const std::string& accessToken) //throw FileError, SysError; returns 0 if not available +{ + //https://developers.google.com/drive/api/v3/reference/about + std::string response; + googleHttpsRequest("/drive/v3/about?fields=storageQuota", { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + if (const JsonValue* storageQuota = getChildFromJsonObject(jresponse, "storageQuota")) + { + const std::optional<std::string> limit = getPrimitiveFromJsonObject(*storageQuota, "limit"); + const std::optional<std::string> usage = getPrimitiveFromJsonObject(*storageQuota, "usage"); + + if (!limit) //"will not be present if the user has unlimited storage." + return 0; + if (usage) + { + const auto usageInt = stringTo<int64_t>(*usage); + const auto limitInt = stringTo<int64_t>(*limit); + + if (0 <= usageInt && usageInt <= limitInt) + return limitInt - usageInt; + } + } + throw SysError(formatGoogleErrorRaw(response)); +} + + +struct GoogleItemDetails +{ + std::string itemName; + bool isFolder = false; + uint64_t fileSize = 0; + time_t modTime = 0; + std::vector<std::string> parentIds; +}; +bool operator==(const GoogleItemDetails& lhs, const GoogleItemDetails& rhs) +{ + return lhs.itemName == rhs.itemName && + lhs.isFolder == rhs.isFolder && + lhs.fileSize == rhs.fileSize && + lhs.modTime == rhs.modTime && + lhs.parentIds == rhs.parentIds; +} + +struct GoogleFileItem +{ + std::string itemId; + GoogleItemDetails details; +}; +std::vector<GoogleFileItem> readFolderContent(const std::string& folderId, //throw FileError + const std::string& accessToken, const GdrivePath& gdrivePath) +{ + + warn_static("perf: trashed=false and ('114231411234' in parents or '123123' in parents)") + + //https://developers.google.com/drive/api/v3/reference/files/list + std::vector<GoogleFileItem> childItems; + try + { + std::optional<std::string> nextPageToken; + do + { + std::string queryParams = xWwwFormUrlEncode( + { + { "spaces", "drive" }, // + { "corpora", "user" }, //"The 'user' corpus includes all files in "My Drive" and "Shared with me" https://developers.google.com/drive/api/v3/about-organization + { "pageSize", "1000" }, //"[1, 1000] Default: 100" + { "fields", "nextPageToken,incompleteSearch,files(name,id,mimeType,size,modifiedTime,parents)" }, //https://developers.google.com/drive/api/v3/reference/files + { "q", "trashed=false and '" + folderId + "' in parents" }, + //{ "q", "sharedWithMe" }, + }); + if (nextPageToken) + queryParams += '&' + xWwwFormUrlEncode({ { "pageToken", *nextPageToken } }); + + std::string response; + googleHttpsRequest("/drive/v3/files?" + queryParams, { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + /**/ nextPageToken = getPrimitiveFromJsonObject(jresponse, "nextPageToken"); + const std::optional<std::string> incompleteSearch = getPrimitiveFromJsonObject(jresponse, "incompleteSearch"); + const JsonValue* files = getChildFromJsonObject (jresponse, "files"); + if (!incompleteSearch || *incompleteSearch != "false" || !files || files->type != JsonValue::Type::array) + throw SysError(formatGoogleErrorRaw(response)); + + for (const auto& childVal : files->arrayVal) + { + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(*childVal, "id"); + const std::optional<std::string> itemName = getPrimitiveFromJsonObject(*childVal, "name"); + const std::optional<std::string> mimeType = getPrimitiveFromJsonObject(*childVal, "mimeType"); + const std::optional<std::string> size = getPrimitiveFromJsonObject(*childVal, "size"); + const std::optional<std::string> modifiedTime = getPrimitiveFromJsonObject(*childVal, "modifiedTime"); + const JsonValue* parents = getChildFromJsonObject (*childVal, "parents"); + + if (!itemId || !itemName || !mimeType || !modifiedTime || !parents) + throw SysError(formatGoogleErrorRaw(response)); + + const bool isFolder = *mimeType == googleFolderMimeType; + const uint64_t fileSize = size ? stringTo<uint64_t>(*size) : 0; //not available for folders + + //RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + const time_t modTime = utcToTimeT(parseTime("%Y-%m-%dT%H:%M:%S", beforeLast(*modifiedTime, '.', IF_MISSING_RETURN_ALL))); //returns -1 on error + if (modTime == -1 || !endsWith(*modifiedTime, 'Z')) //'Z' means "UTC" => it seems Google doesn't use the time-zone offset postfix + throw SysError(L"Modification time could not be parsed. (" + utfTo<std::wstring>(*modifiedTime) + L")"); + + std::vector<std::string> parentIds; + for (const auto& parentVal : parents->arrayVal) + { + if (parentVal->type != JsonValue::Type::string) + throw SysError(formatGoogleErrorRaw(response)); + parentIds.push_back(parentVal->primVal); + } + assert(std::find(parentIds.begin(), parentIds.end(), folderId) != parentIds.end()); + + childItems.push_back({ *itemId, { *itemName, isFolder, fileSize, modTime, std::move(parentIds) } }); + } + } + while (nextPageToken); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read directory %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); } + + return childItems; +} + + +struct ChangeItem +{ + std::string itemId; + std::optional<GoogleItemDetails> details; //empty if item was deleted! +}; +struct ChangesDelta +{ + std::string newStartPageToken; + std::vector<ChangeItem> changes; +}; +ChangesDelta getChangesDelta(const std::string& startPageToken, //throw FileError + const std::string& accessToken, const Zstring& googleUserEmail) +{ + try //https://developers.google.com/drive/api/v3/reference/changes/list + { + ChangesDelta delta; + std::optional<std::string> nextPageToken = startPageToken; + for (;;) + { + std::string queryParams = xWwwFormUrlEncode( + { + { "pageToken", *nextPageToken }, + { "pageSize", "1000" }, //"[1, 1000] Default: 100" + { "restrictToMyDrive", "true" }, //important! otherwise we won't get "removed: true" (because file may still be accessible from other Corpora) + { "spaces", "drive" }, + { "fields", "kind,nextPageToken,newStartPageToken,changes(kind,removed,fileId,file(name,mimeType,size,modifiedTime,parents,trashed))" }, + }); + + std::string response; + googleHttpsRequest("/drive/v3/changes?" + queryParams, { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + /**/ nextPageToken = getPrimitiveFromJsonObject(jresponse, "nextPageToken"); + const std::optional<std::string> newStartPageToken = getPrimitiveFromJsonObject(jresponse, "newStartPageToken"); + const std::optional<std::string> listKind = getPrimitiveFromJsonObject(jresponse, "kind"); + const JsonValue* changes = getChildFromJsonObject (jresponse, "changes"); + + if (!!nextPageToken == !!newStartPageToken || //there can be only one + !listKind || *listKind != "drive#changeList" || + !changes || changes->type != JsonValue::Type::array) + throw SysError(formatGoogleErrorRaw(response)); + + for (const auto& childVal : changes->arrayVal) + { + const std::optional<std::string> kind = getPrimitiveFromJsonObject(*childVal, "kind"); + const std::optional<std::string> removed = getPrimitiveFromJsonObject(*childVal, "removed"); + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(*childVal, "fileId"); + const JsonValue* file = getChildFromJsonObject (*childVal, "file"); + if (!kind || *kind != "drive#change" || !removed || !itemId) + throw SysError(formatGoogleErrorRaw(response)); + + ChangeItem changeItem; + changeItem.itemId = *itemId; + if (*removed != "true") + { + if (!file) + throw SysError(formatGoogleErrorRaw(response)); + + const std::optional<std::string> itemName = getPrimitiveFromJsonObject(*file, "name"); + const std::optional<std::string> mimeType = getPrimitiveFromJsonObject(*file, "mimeType"); + const std::optional<std::string> size = getPrimitiveFromJsonObject(*file, "size"); + const std::optional<std::string> modifiedTime = getPrimitiveFromJsonObject(*file, "modifiedTime"); + const std::optional<std::string> trashed = getPrimitiveFromJsonObject(*file, "trashed"); + const JsonValue* parents = getChildFromJsonObject (*file, "parents"); + if (!itemName || !mimeType || !modifiedTime || !trashed || !parents) + throw SysError(formatGoogleErrorRaw(response)); + + if (*trashed != "true") + { + GoogleItemDetails itemDetails = {}; + itemDetails.itemName = *itemName; + itemDetails.isFolder = *mimeType == googleFolderMimeType; + itemDetails.fileSize = size ? stringTo<uint64_t>(*size) : 0; //not available for folders + + //RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + itemDetails.modTime = utcToTimeT(parseTime("%Y-%m-%dT%H:%M:%S", beforeLast(*modifiedTime, '.', IF_MISSING_RETURN_ALL))); //returns -1 on error + if (itemDetails.modTime == -1 || !endsWith(*modifiedTime, 'Z')) //'Z' means "UTC" => it seems Google doesn't use the time-zone offset postfix + throw SysError(L"Modification time could not be parsed. (" + utfTo<std::wstring>(*modifiedTime) + L")"); + + for (const auto& parentVal : parents->arrayVal) + { + if (parentVal->type != JsonValue::Type::string) + throw SysError(formatGoogleErrorRaw(response)); + itemDetails.parentIds.push_back(parentVal->primVal); + } + changeItem.details = std::move(itemDetails); + } + } + delta.changes.push_back(std::move(changeItem)); + } + + if (!nextPageToken) + { + delta.newStartPageToken = *newStartPageToken; + return delta; + } + } + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot monitor directory %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), e.toString()); } +} + + +std::string /*startPageToken*/ getChangesCurrentToken(const std::string& accessToken, const Zstring& googleUserEmail) //throw FileError +{ + //https://developers.google.com/drive/api/v3/reference/changes/getStartPageToken + std::string response; + googleHttpsRequest("/drive/v3/changes/startPageToken", { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> startPageToken = getPrimitiveFromJsonObject(jresponse, "startPageToken"); + if (!startPageToken) + throw FileError(replaceCpy(_("Cannot monitor directory %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), formatGoogleErrorRaw(response)); + + return *startPageToken; +} + + +//- if item is a folder: deletes recursively!!! +//- even deletes a hardlink with multiple parents => use gdriveUnlinkParent() first +void gdriveDeleteItem(const std::string& itemId, const std::string& accessToken) //throw FileError, SysError +{ + //https://developers.google.com/drive/api/v3/reference/files/delete + std::string response; + const HttpSession::HttpResult httpResult = googleHttpsRequest("/drive/v3/files/" + itemId, { "Authorization: Bearer " + accessToken }, //throw FileError + { { CURLOPT_CUSTOMREQUEST, "DELETE" } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + //"If successful, this method returns an empty response body" + if (!response.empty() || httpResult.statusCode != 204) + throw SysError(formatGoogleErrorRaw(response)); +} + + +//item is NOT deleted when last parent is removed: it is just not accessible via the "My Drive" hierarchy but still adds to quota! => use for hard links only! +void gdriveUnlinkParent(const std::string& itemId, const std::string& parentFolderId, const std::string& accessToken) //throw FileError, SysError +{ + //https://developers.google.com/drive/api/v3/reference/files/update + const std::string queryParams = xWwwFormUrlEncode( + { + { "removeParents", parentFolderId }, + { "fields", "id,parents"}, //for test if operation was successful + }); + std::string response; + googleHttpsRequest("/drive/v3/files/" + itemId + '?' + queryParams, //throw FileError + { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_CUSTOMREQUEST, "PATCH" }, { CURLOPT_POSTFIELDS, "{}" } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); /*throw JsonParsingError*/ } + catch (const JsonParsingError&) {} + + const std::optional<std::string> id = getPrimitiveFromJsonObject(jresponse, "id"); //id is returned on "success", unlike "parents", see below... + const JsonValue* parents = getChildFromJsonObject(jresponse, "parents"); + if (!id || *id != itemId) + throw SysError(formatGoogleErrorRaw(response)); + + if (parents) //when last parent is removed (=> Google deletes item permanently), Google does NOT return the parents array (not even an empty one!) + if (parents->type != JsonValue::Type::array || + std::any_of(parents->arrayVal.begin(), parents->arrayVal.end(), + [&](const std::unique_ptr<JsonValue>& jval) { return jval->type == JsonValue::Type::string && jval->primVal == parentFolderId; })) + throw SysError(L"Google Drive internal failure"); //user should never see this... +} + + +//- if item is a folder: trashes recursively!!! +//- a hardlink with multiple parents will be not be accessible anymore via any of its path aliases! +void gdriveMoveToTrash(const std::string& itemId, const std::string& accessToken) //throw FileError, SysError +{ + //https://developers.google.com/drive/api/v3/reference/files/update + const std::string postBuf = R"({ "trashed": true })"; + + std::string response; + googleHttpsRequest("/drive/v3/files/" + itemId + "?fields=trashed", //throw FileError + { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_CUSTOMREQUEST, "PATCH" }, { CURLOPT_POSTFIELDS, postBuf.c_str() } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); /*throw JsonParsingError*/ } + catch (const JsonParsingError&) {} + + const std::optional<std::string> trashed = getPrimitiveFromJsonObject(jresponse, "trashed"); + if (!trashed || *trashed != "true") + throw SysError(formatGoogleErrorRaw(response)); +} + + +//folder name already existing? will (happily) create duplicate folders => caller must check! +std::string /*folderId*/ gdriveCreateFolderPlain(const Zstring& folderName, const std::string& parentFolderId, const std::string& accessToken) //throw FileError, SysError +{ + //https://developers.google.com/drive/api/v3/folder#creating_a_folder + std::string postBuf = "{\n"; + postBuf += "\"mimeType\": \"" + std::string(googleFolderMimeType) + "\",\n"; + postBuf += "\"name\": \"" + utfTo<std::string>(folderName) + "\",\n"; + postBuf += "\"parents\": [\"" + parentFolderId + "\"]\n"; //[!] no trailing comma! + postBuf += "}"; + + std::string response; + googleHttpsRequest("/drive/v3/files?fields=id", { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_POSTFIELDS, postBuf.c_str() } }, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(jresponse, "id"); + if (!itemId) + throw SysError(formatGoogleErrorRaw(response)); + return *itemId; +} + + +//target name already existing? will (happily) create duplicate items => caller must check! +void gdriveMoveAndRenameItem(const std::string& itemId, const std::string& parentIdOld, const std::string& parentIdNew, + const Zstring& newName, time_t newModTime, const std::string& accessToken) //throw FileError, SysError +{ + //https://developers.google.com/drive/api/v3/folder#moving_files_between_folders + std::string queryParams = "fields=name,parents"; //for test if operation was successful + + if (parentIdOld != parentIdNew) + queryParams += '&' + xWwwFormUrlEncode( + { + { "removeParents", parentIdOld }, + { "addParents", parentIdNew }, + }); + + //more Google Drive peculiarities: changing the file name changes modifiedTime!!! => workaround: + + //RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + const std::string dateTime = formatTime<std::string>("%Y-%m-%dT%H:%M:%S.000Z", getUtcTime(newModTime)); //returns empty string on failure + if (dateTime.empty()) + throw SysError(L"Invalid modification time (time_t: " + numberTo<std::wstring>(newModTime) + L")"); + + std::string postBuf = "{\n"; + //postBuf += "\"name\": \"" + utfTo<std::string>(newName) + "\"\n"; + postBuf += "\"name\": \"" + utfTo<std::string>(newName) + "\",\n"; + postBuf += "\"modifiedTime\": \"" + dateTime + "\"\n"; //[!] no trailing comma! + postBuf += "}"; + + std::string response; + googleHttpsRequest("/drive/v3/files/" + itemId + '?' + queryParams, //throw FileError + { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_CUSTOMREQUEST, "PATCH" }, { CURLOPT_POSTFIELDS, postBuf.c_str() } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); /*throw JsonParsingError*/ } + catch (const JsonParsingError&) {} + + const std::optional<std::string> name = getPrimitiveFromJsonObject(jresponse, "name"); + const JsonValue* parents = getChildFromJsonObject(jresponse, "parents"); + if (!name || *name != utfTo<std::string>(newName) || + !parents || parents->type != JsonValue::Type::array) + throw SysError(formatGoogleErrorRaw(response)); + + if (!std::any_of(parents->arrayVal.begin(), parents->arrayVal.end(), + [&](const std::unique_ptr<JsonValue>& jval) { return jval->type == JsonValue::Type::string && jval->primVal == parentIdNew; })) + throw SysError(L"Google Drive internal failure"); //user should never see this... +} + + +#if 0 +void setModTime(const std::string& itemId, time_t modTime, //throw FileError + const std::string& accessToken, const GdrivePath& gdrivePath) +{ + try //https://developers.google.com/drive/api/v3/reference/files/update + { + //RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + const std::string dateTime = formatTime<std::string>("%Y-%m-%dT%H:%M:%S.000Z", getUtcTime(modTime)); //returns empty string on failure + if (dateTime.empty()) + throw SysError(L"Invalid modification time (time_t: " + numberTo<std::wstring>(modTime) + L")"); + + const std::string postBuf = R"({ "modifiedTime": ")" + dateTime + "\" }"; + + std::string response; + googleHttpsRequest("/drive/v3/files/" + itemId + "?fields=modifiedTime", //throw FileError + { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_CUSTOMREQUEST, "PATCH" }, { CURLOPT_POSTFIELDS, postBuf.c_str() } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); /*throw JsonParsingError*/ } + catch (const JsonParsingError&) {} + + const std::optional<std::string> modifiedTime = getPrimitiveFromJsonObject(jresponse, "modifiedTime"); + if (!modifiedTime || *modifiedTime != dateTime) + throw SysError(formatGoogleErrorRaw(response)); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot write modification time of %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); + } +} +#endif + + +void gdriveDownloadFile(const std::string& itemId, const std::function<void(const void* buffer, size_t bytesToWrite)>& writeBlock /*throw X*/, //throw FileError, X + const std::string& accessToken, const GdrivePath& gdrivePath) +{ + //https://developers.google.com/drive/api/v3/manage-downloads + std::string response; + const HttpSession::HttpResult httpResult = googleHttpsRequest("/drive/v3/files/" + itemId + "?alt=media", //throw FileError, X + { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, + [&](const void* buffer, size_t bytesToWrite) + { + writeBlock(buffer, bytesToWrite); //throw X + if (response.size() < 10000) //always save front part of the response in case we get an error + response.append(static_cast<const char*>(buffer), bytesToWrite); + }, nullptr /*readRequest*/); + + if (httpResult.statusCode / 100 != 2) + throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), formatGoogleErrorRaw(response)); +} + + +#if 0 +//file name already existing? => duplicate file created! +//note: Google Drive upload is already transactional! +//upload "small files" (5 MB or less; enforced by Google?) in a single round-trip +std::string /*itemId*/ gdriveUploadSmallFile(const Zstring& fileName, const std::string& parentFolderId, uint64_t streamSize, std::optional<time_t> modTime, //throw FileError, X + const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/, //returning 0 signals EOF: Posix read() semantics + const std::string& accessToken, const GdrivePath& gdrivePath) +{ + //https://developers.google.com/drive/api/v3/folder#inserting_a_file_in_a_folder + //https://developers.google.com/drive/api/v3/multipart-upload + + std::string metaDataBuf = "{\n"; + if (modTime) //convert to RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + { + const std::string dateTime = formatTime<std::string>("%Y-%m-%dT%H:%M:%S.000Z", getUtcTime(*modTime)); //returns empty string on failure + if (dateTime.empty()) + throw SysError(L"Invalid modification time (time_t: " + numberTo<std::wstring>(*modTime) + L")"); + + metaDataBuf += "\"modifiedTime\": \"" + dateTime + "\",\n"; + } + metaDataBuf += "\"name\": \"" + utfTo<std::string>(fileName) + "\",\n"; + metaDataBuf += "\"parents\": [\"" + parentFolderId + "\"]\n"; //[!] no trailing comma! + metaDataBuf += "}"; + + //allowed chars for border: DIGIT ALPHA ' ( ) + _ , - . / : = ? + const std::string boundaryString = stringEncodeBase64(generateGUID() + generateGUID()); + + const std::string postBufHead = "--" + boundaryString + "\r\n" + "Content-Type: application/json; charset=UTF-8" "\r\n" + /**/ "\r\n" + + metaDataBuf + "\r\n" + "--" + boundaryString + "\r\n" + "Content-Type: application/octet-stream" "\r\n" + /**/ "\r\n"; + + const std::string postBufTail = "\r\n--" + boundaryString + "--"; + + auto readMultipartBlock = [&, headPos = size_t(0), eof = false, tailPos = size_t(0)](void* buffer, size_t bytesToRead) mutable -> size_t + { + auto it = static_cast<std::byte*>(buffer); + const auto itEnd = it + bytesToRead; + + if (headPos < postBufHead.size()) + { + const size_t junkSize = std::min<ptrdiff_t>(postBufHead.size() - headPos, itEnd - it); + std::memcpy(it, &postBufHead[headPos], junkSize); + headPos += junkSize; + it += junkSize; + } + if (it != itEnd) + { + if (!eof) //don't assume readBlock() will return streamSize bytes as promised => exhaust and let Google Drive fail if there is a mismatch in Content-Length! + { + const size_t junkSize = readBlock(it, itEnd - it); //throw X + it += junkSize; + + if (junkSize != 0) + return it - static_cast<std::byte*>(buffer); //perf: if input stream is at the end, should we immediately append postBufTail (and avoid extra TCP package)? => negligible! + else + eof = true; + } + if (it != itEnd) + if (tailPos < postBufTail.size()) + { + const size_t junkSize = std::min<ptrdiff_t>(postBufTail.size() - tailPos, itEnd - it); + std::memcpy(it, &postBufTail[tailPos], junkSize); + tailPos += junkSize; + it += junkSize; + } + } + return it - static_cast<std::byte*>(buffer); + }; + +TODO: + gzip-compress HTTP request body! + + try + { + std::string response; + const HttpSession::HttpResult httpResult = googleHttpsRequest("/upload/drive/v3/files?uploadType=multipart", //throw FileError, X + { + "Authorization: Bearer " + accessToken, + "Content-Type: multipart/related; boundary=" + boundaryString, + "Content-Length: " + numberTo<std::string>(postBufHead.size() + streamSize + postBufTail.size()) + }, + { { CURLOPT_POST, 1 } }, //otherwise HttpSession::perform() will PUT + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, readMultipartBlock ); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(jresponse, "id"); + if (!itemId) + throw SysError(formatGoogleErrorRaw(response)); + + return *itemId; + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); + } +} +#endif + + +//file name already existing? => duplicate file created! +//note: Google Drive upload is already transactional! +std::string /*itemId*/ gdriveUploadFile(const Zstring& fileName, const std::string& parentFolderId, std::optional<time_t> modTime, //throw FileError, X + const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/, //returning 0 signals EOF: Posix read() semantics + const std::string& accessToken, const GdrivePath& gdrivePath) +{ + //https://developers.google.com/drive/api/v3/folder#inserting_a_file_in_a_folder + //https://developers.google.com/drive/api/v3/resumable-upload + try + { + //step 1: initiate resumable upload session + std::string uploadUrlRelative; + { + std::string postBuf = "{\n"; + if (modTime) //convert to RFC 3339 date-time: e.g. "2018-09-29T08:39:12.053Z" + { + const std::string dateTime = formatTime<std::string>("%Y-%m-%dT%H:%M:%S.000Z", getUtcTime(*modTime)); //returns empty string on failure + if (dateTime.empty()) + throw SysError(L"Invalid modification time (time_t: " + numberTo<std::wstring>(*modTime) + L")"); + + postBuf += "\"modifiedTime\": \"" + dateTime + "\",\n"; + } + postBuf += "\"name\": \"" + utfTo<std::string>(fileName) + "\",\n"; + postBuf += "\"parents\": [\"" + parentFolderId + "\"]\n"; //[!] no trailing comma! + postBuf += "}"; + + std::string uploadUrl; + + auto onBytesReceived = [&](const void* buffer, size_t len) + { + //inside libcurl's C callstack => better not throw exceptions here!!! + //"The callback will be called once for each header and only complete header lines are passed on to the callback" (including \r\n at the end) + const auto strBegin = static_cast<const char*>(buffer); + if (startsWithAsciiNoCase(StringRef<const char>(strBegin, strBegin + len), "Location:")) + { + uploadUrl.assign(strBegin, len); //not null-terminated! + uploadUrl = afterFirst(uploadUrl, ':', IF_MISSING_RETURN_NONE); + trim(uploadUrl); + } + return len; + }; + using ReadCbType = decltype(onBytesReceived); + using ReadCbWrapperType = size_t (*)(const void* buffer, size_t size, size_t nitems, void* callbackData); //needed for cdecl function pointer cast + ReadCbWrapperType onBytesReceivedWrapper = [](const void* buffer, size_t size, size_t nitems, void* callbackData) + { + auto cb = static_cast<ReadCbType*>(callbackData); //free this poor little C-API from its shackles and redirect to a proper lambda + return (*cb)(buffer, size * nitems); + }; + + std::string response; + const HttpSession::HttpResult httpResult = googleHttpsRequest("/upload/drive/v3/files?uploadType=resumable", //throw FileError + { "Authorization: Bearer " + accessToken, "Content-Type: application/json; charset=UTF-8" }, + { { CURLOPT_POSTFIELDS, postBuf.c_str() }, { CURLOPT_HEADERDATA, &onBytesReceived }, { CURLOPT_HEADERFUNCTION, onBytesReceivedWrapper } }, + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + if (httpResult.statusCode != 200) + throw SysError(formatGoogleErrorRaw(response)); + + if (!startsWith(uploadUrl, "https://www.googleapis.com/")) + throw SysError(L"Invalid upload URL: " + utfTo<std::wstring>(uploadUrl)); //user should never see this + + uploadUrlRelative = afterFirst(uploadUrl, "googleapis.com", IF_MISSING_RETURN_NONE); + } + //--------------------------------------------------- + //step 2: upload file content + + //not officially documented, but Google Drive supports compressed file upload when "Content-Encoding: gzip" is set! :))) + InputStreamAsGzip gzipStream(readBlock); //throw ZlibInternalError + + auto readBlockAsGzip = [&](void* buffer, size_t bytesToRead) { return gzipStream.read(buffer, bytesToRead); }; //throw ZlibInternalError, X; + //returns "bytesToRead" bytes unless end of stream! => fits into "0 signals EOF: Posix read() semantics" + + std::string response; + googleHttpsRequest(uploadUrlRelative, { "Content-Encoding: gzip" }, {} /*extraOptions*/, //throw FileError, X + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, readBlockAsGzip); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(jresponse, "id"); + if (!itemId) + throw SysError(formatGoogleErrorRaw(response)); + + return *itemId; + } + catch (ZlibInternalError&) + { + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), L"zlib internal error"); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); + } +} + + +//instead of the "root" alias Google uses an actual ID in file metadata +std::string /*itemId*/ getRootItemId(const std::string& accessToken, const Zstring& googleUserEmail) //throw FileError +{ + //https://developers.google.com/drive/api/v3/reference/files/get + std::string response; + googleHttpsRequest("/drive/v3/files/root?fields=id", { "Authorization: Bearer " + accessToken }, {} /*extraOptions*/, //throw FileError + [&](const void* buffer, size_t bytesToWrite) { response.append(static_cast<const char*>(buffer), bytesToWrite); }, nullptr /*readRequest*/); + + JsonValue jresponse; + try { jresponse = parseJson(response); } + catch (JsonParsingError&) {} + + const std::optional<std::string> itemId = getPrimitiveFromJsonObject(jresponse, "id"); + if (!itemId) + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), formatGoogleErrorRaw(response)); + + return *itemId; +} + + +class GoogleAccessBuffer //per-user-session! => serialize access (perf: amortized fully buffered!) +{ +public: + GoogleAccessBuffer(const GoogleAccessInfo& accessInfo) : accessInfo_(accessInfo) {} + + GoogleAccessBuffer(MemoryStreamIn<ByteArray>& stream) //throw UnexpectedEndOfStreamError + { + accessInfo_.accessToken.validUntil = readNumber<int64_t>(stream); // + accessInfo_.accessToken.value = readContainer<std::string>(stream); // + accessInfo_.refreshToken = readContainer<std::string>(stream); //UnexpectedEndOfStreamError + accessInfo_.userInfo.displayName = utfTo<std::wstring>(readContainer<std::string>(stream)); // + accessInfo_.userInfo.email = utfTo< Zstring>(readContainer<std::string>(stream)); // + } + + void serialize(MemoryStreamOut<ByteArray>& stream) const + { + writeNumber<int64_t>(stream, accessInfo_.accessToken.validUntil); + static_assert(sizeof(accessInfo_.accessToken.validUntil) <= sizeof(int64_t)); //ensure cross-platform compatibility! + writeContainer(stream, accessInfo_.accessToken.value); + writeContainer(stream, accessInfo_.refreshToken); + writeContainer(stream, utfTo<std::string>(accessInfo_.userInfo.displayName)); + writeContainer(stream, utfTo<std::string>(accessInfo_.userInfo.email)); + } + + std::string getAccessToken() //throw FileError + { + if (accessInfo_.accessToken.validUntil <= std::time(nullptr) + std::chrono::seconds(HTTP_SESSION_ACCESS_TIME_OUT).count() + 5 /*some leeway*/) //expired/will expire + accessInfo_.accessToken = refreshAccessToGoogleDrive(accessInfo_.refreshToken, accessInfo_.userInfo.email); //throw FileError + + assert(accessInfo_.accessToken.validUntil > std::time(nullptr) + std::chrono::seconds(HTTP_SESSION_ACCESS_TIME_OUT).count()); + return accessInfo_.accessToken.value; + } + + //const std::wstring& getUserDisplayName() const { return accessInfo_.userInfo.displayName; } + const Zstring& getUserEmail() const { return accessInfo_.userInfo.email; } + + void update(const GoogleAccessInfo& accessInfo) + { + if (!equalAsciiNoCase(accessInfo.userInfo.email, accessInfo_.userInfo.email)) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + accessInfo_ = accessInfo; + } + +private: + GoogleAccessBuffer (const GoogleAccessBuffer&) = delete; + GoogleAccessBuffer& operator=(const GoogleAccessBuffer&) = delete; + + GoogleAccessInfo accessInfo_; +}; + + +class GooglePersistentSessions; + + +class GoogleFileState //per-user-session! => serialize access (perf: amortized fully buffered!) +{ +public: + GoogleFileState(GoogleAccessBuffer& accessBuf) : accessBuf_(accessBuf) //throw FileError + { + lastSyncToken_ = getChangesCurrentToken(accessBuf_.getAccessToken(), accessBuf_.getUserEmail()); //throw FileError + rootId_ = getRootItemId (accessBuf_.getAccessToken(), accessBuf_.getUserEmail()); //throw FileError + } + + GoogleFileState(MemoryStreamIn<ByteArray>& stream, GoogleAccessBuffer& accessBuf) : accessBuf_(accessBuf) //throw UnexpectedEndOfStreamError + { + lastSyncToken_ = readContainer<std::string>(stream); //UnexpectedEndOfStreamError + rootId_ = readContainer<std::string>(stream); //UnexpectedEndOfStreamError + + for (;;) + { + const std::string folderId = readContainer<std::string>(stream); //UnexpectedEndOfStreamError + if (folderId.empty()) + break; + folderContents_[folderId].isKnownFolder = true; + } + + size_t itemCount = readNumber<int32_t>(stream); + while (itemCount-- != 0) + { + const std::string itemId = readContainer<std::string>(stream); //UnexpectedEndOfStreamError + + GoogleItemDetails details = {}; + details.itemName = readContainer<std::string>(stream); // + details.isFolder = readNumber <int8_t>(stream) != 0; //UnexpectedEndOfStreamError + details.fileSize = readNumber <uint64_t>(stream); // + details.modTime = readNumber <int64_t>(stream); // + + size_t parentsCount = readNumber<int32_t>(stream); //UnexpectedEndOfStreamError + while (parentsCount-- != 0) + details.parentIds.push_back(readContainer<std::string>(stream)); //UnexpectedEndOfStreamError + + updateItemState(itemId, std::move(details)); + } + } + + void serialize(MemoryStreamOut<ByteArray>& stream) const + { + writeContainer(stream, lastSyncToken_); + writeContainer(stream, rootId_); + + for (const auto& [folderId, content] : folderContents_) + if (folderId.empty()) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + else if (content.isKnownFolder) + writeContainer(stream, folderId); + writeContainer(stream, std::string()); //sentinel + + writeNumber(stream, static_cast<int32_t>(itemDetails_.size())); + for (const auto& [itemId, details] : itemDetails_) + { + writeContainer(stream, itemId); + writeContainer(stream, details.itemName); + writeNumber< int8_t>(stream, details.isFolder); + writeNumber<uint64_t>(stream, details.fileSize); + writeNumber< int64_t>(stream, details.modTime); + static_assert(sizeof(details.modTime) <= sizeof(int64_t)); //ensure cross-platform compatibility! + + writeNumber(stream, static_cast<int32_t>(details.parentIds.size())); + for (const std::string& parentId : details.parentIds) + writeContainer(stream, parentId); + } + } + + struct PathStatus + { + std::string existingItemId; + bool existingIsFolder = false; + AfsPath existingPath; //input path =: existingPath + relPath + std::vector<Zstring> relPath; // + }; + PathStatus getPathStatus(const AfsPath& afsPath) //throw SysError + { + const std::vector<Zstring> relPath = split(afsPath.value, FILE_NAME_SEPARATOR, SplitType::SKIP_EMPTY); + if (relPath.empty()) + return { rootId_, true /*existingIsFolder*/, AfsPath(), {} }; + + return getPathStatusSub(rootId_, AfsPath(), relPath); //throw SysError + } + + std::string /*itemId*/ getItemId(const AfsPath& afsPath) //throw SysError + { + const GoogleFileState::PathStatus ps = getPathStatus(afsPath); //throw SysError + if (ps.relPath.empty()) + return ps.existingItemId; + + const AfsPath afsPathMissingChild(nativeAppendPaths(ps.existingPath.value, ps.relPath.front())); + throw SysError(replaceCpy(_("Cannot find %x."), L"%x", fmtPath(getGoogleDisplayPath({ accessBuf_.getUserEmail(), afsPathMissingChild })))); + } + + std::pair<std::string /*itemId*/, GoogleItemDetails> getFileAttributes(const AfsPath& afsPath) //throw SysError + { + const std::string fileId = getItemId(afsPath); //throw SysError + auto it = itemDetails_.find(fileId); + if (it == itemDetails_.end()) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + return *it; + } + + std::optional<std::vector<GoogleFileItem>> tryGetBufferedFolderContent(const std::string& folderId) const + { + auto it = folderContents_.find(folderId); + if (it == folderContents_.end() || !it->second.isKnownFolder) + return std::nullopt; + + std::vector<GoogleFileItem> childItems; + for (auto itChild : it->second.childItems) + { + const auto& [childId, childDetails] = *itChild; + childItems.push_back({ childId, childDetails }); + } + return std::move(childItems); //[!] need std::move! + } + + //-------------- notifications -------------- + using ItemIdDelta = std::unordered_set<std::string>; + + struct FileStateDelta //as long as instance exists, GoogleFileItem will log all changed items + { + FileStateDelta() {} + private: + FileStateDelta(const std::shared_ptr<const ItemIdDelta>& cids) : changedIds(cids) {} + friend class GoogleFileState; + std::shared_ptr<const ItemIdDelta> changedIds; //lifetime is managed by caller; access *only* by GoogleFileState! + }; + + void notifyFolderContent(const FileStateDelta& stateDelta, const std::string& folderId, const std::vector<GoogleFileItem>& childItems) + { + folderContents_[folderId].isKnownFolder = true; + + for (const GoogleFileItem& item : childItems) + notifyItemUpdate(stateDelta, item.itemId, item.details); + + //- should we remove parent links for items that are not children of folderId anymore (as of this update)?? => fringe case during first update! (still: maybe trigger sync?) + //- what if there are multiple folder state updates incoming in wrong order!? => notifyItemUpdate() will sort it out! + } + + void notifyItemCreated(const FileStateDelta& stateDelta, const GoogleFileItem& item) + { + notifyItemUpdate(stateDelta, item.itemId, item.details); + } + + void notifyFolderCreated(const FileStateDelta& stateDelta, const std::string& folderId, const Zstring& folderName, const std::string& parentId) + { + GoogleItemDetails details = {}; + details.itemName = utfTo<std::string>(folderName); + details.isFolder = true; + details.modTime = std::time(nullptr); + details.parentIds.push_back(parentId); + + //avoid needless conflicts due to different Google Drive folder modTime! + auto it = itemDetails_.find(folderId); + if (it != itemDetails_.end()) + details.modTime = it->second.modTime; + + notifyItemUpdate(stateDelta, folderId, details); + } + + void notifyItemDeleted(const FileStateDelta& stateDelta, const std::string& itemId) + { + notifyItemUpdate(stateDelta, itemId, std::nullopt); + } + + void notifyParentRemoved(const FileStateDelta& stateDelta, const std::string& itemId, const std::string& parentIdOld) + { + auto it = itemDetails_.find(itemId); + if (it != itemDetails_.end()) + { + GoogleItemDetails detailsNew = it->second; + eraseIf(detailsNew.parentIds, [&](const std::string& id) { return id == parentIdOld; }); + notifyItemUpdate(stateDelta, itemId, detailsNew); + } + else //conflict!!! + markSyncDue(); + } + + void notifyMoveAndRename(const FileStateDelta& stateDelta, const std::string& itemId, const std::string& parentIdOld, const std::string& parentIdNew, const Zstring& newName) + { + auto it = itemDetails_.find(itemId); + if (it != itemDetails_.end()) + { + GoogleItemDetails detailsNew = it->second; + detailsNew.itemName = utfTo<std::string>(newName); + + eraseIf(detailsNew.parentIds, [&](const std::string& id) { return id == parentIdOld || id == parentIdNew; }); // + detailsNew.parentIds.push_back(parentIdNew); //not a duplicate + + notifyItemUpdate(stateDelta, itemId, detailsNew); + } + else //conflict!!! + markSyncDue(); + } + +private: + GoogleFileState (const GoogleFileState&) = delete; + GoogleFileState& operator=(const GoogleFileState&) = delete; + + friend class GooglePersistentSessions; + + void notifyItemUpdate(const FileStateDelta& stateDelta, const std::string& itemId, const std::optional<GoogleItemDetails>& details) + { + if (stateDelta.changedIds->find(itemId) == stateDelta.changedIds->end()) //=> no conflicting changes in the meantime + updateItemState(itemId, details); //accept new state data + else //conflict? + { + auto it = itemDetails_.find(itemId); + if (!details == (it == itemDetails_.end())) + if (!details || *details == it->second) + return; //notified changes match our current file state + //else: conflict!!! unclear which has the more recent data! + markSyncDue(); + } + } + + FileStateDelta registerFileStateDelta() + { + auto deltaPtr = std::make_shared<ItemIdDelta>(); + changeLog_.push_back(deltaPtr); + return FileStateDelta(deltaPtr); + } + + bool syncIsDue() const { return std::chrono::steady_clock::now() >= lastSyncTime_ + GOOGLE_DRIVE_SYNC_INTERVAL; } + + void markSyncDue() { lastSyncTime_ = std::chrono::steady_clock::now() - GOOGLE_DRIVE_SYNC_INTERVAL; } + + + void syncWithGoogle() //throw FileError + { + const ChangesDelta delta = getChangesDelta(lastSyncToken_, accessBuf_.getAccessToken(), accessBuf_.getUserEmail()); //throw FileError + + for (const ChangeItem& item : delta.changes) + updateItemState(item.itemId, item.details); + + lastSyncToken_ = delta.newStartPageToken; + lastSyncTime_ = std::chrono::steady_clock::now(); + + //good to know: if item is created and deleted between polling for changes it is still reported as deleted by Google! + //Same goes for any other change that is undone in between change notification syncs. + } + + PathStatus getPathStatusSub(const std::string& folderId, const AfsPath& folderPath, const std::vector<Zstring>& relPath) //throw SysError + { + assert(!relPath.empty()); + + std::vector<DetailsIterator>* childItems = nullptr; + auto itKnown = folderContents_.find(folderId); + if (itKnown != folderContents_.end() && itKnown->second.isKnownFolder) + childItems = &(itKnown->second.childItems); + else + { + try + { + notifyFolderContent(registerFileStateDelta(), folderId, readFolderContent(folderId, accessBuf_.getAccessToken(), { accessBuf_.getUserEmail(), folderPath })); //throw FileError + } + catch (const FileError& e) { throw SysError(e.toString()); } //path-resolution errors should be further enriched by context info => SysError + + if (!folderContents_[folderId].isKnownFolder) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + childItems = &folderContents_[folderId].childItems; + } + + auto itFound = itemDetails_.cend(); + for (const DetailsIterator& itDetails : *childItems) + //Since Google Drive has no concept of a file path, we have to roll our own "path to id" mapping => let's use the platform-native style + if (equalNativePath(utfTo<Zstring>(itDetails->second.itemName), relPath.front())) + { + if (itFound != itemDetails_.end()) + throw SysError(replaceCpy(_("Cannot find %x."), L"%x", + fmtPath(getGoogleDisplayPath({ accessBuf_.getUserEmail(), AfsPath(nativeAppendPaths(folderPath.value, relPath.front())) }))) + L" " + + replaceCpy(_("The name %x is used by more than one item in the folder."), L"%x", fmtPath(relPath.front()))); + + itFound = itDetails; + } + + if (itFound == itemDetails_.end()) + return { folderId, true /*existingIsFolder*/, folderPath, relPath }; //always a folder, see check before recursion above + else + { + const auto& [childId, childDetails] = *itFound; + const AfsPath childItemPath(nativeAppendPaths(folderPath.value, relPath.front())); + const std::vector<Zstring> childRelPath(relPath.begin() + 1, relPath.end()); + + if (childRelPath.empty() || !childDetails.isFolder /*obscure, but possible (and not an error)*/) + return { childId, childDetails.isFolder, childItemPath, childRelPath }; + + return getPathStatusSub(childId, childItemPath, childRelPath); //throw SysError + } + } + + void updateItemState(const std::string& itemId, const std::optional<GoogleItemDetails>& details) + { + auto it = itemDetails_.find(itemId); + + if (!details == (it == itemDetails_.end())) + if (!details || *details == it->second) //notified changes match our current file state + return; //=> avoid misleading changeLog_ entries after Google Drive sync!!! + + //update change logs (and clean up obsolete entries) + eraseIf(changeLog_, [&](std::weak_ptr<ItemIdDelta>& weakPtr) + { + if (std::shared_ptr<ItemIdDelta> iid = weakPtr.lock()) + { + (*iid).insert(itemId); + return false; + } + else + return true; + }); + + //update file state + if (details) + { + if (it != itemDetails_.end()) //update + { + if (it->second.isFolder != details->isFolder) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); //WTF!? + + std::vector<std::string> parentIdsNew = details->parentIds; + std::vector<std::string> parentIdsRemoved = it->second.parentIds; + eraseIf(parentIdsNew, [&](const std::string& id) { return std::find(it->second.parentIds.begin(), it->second.parentIds.end(), id) != it->second.parentIds.end(); }); + eraseIf(parentIdsRemoved, [&](const std::string& id) { return std::find(details->parentIds.begin(), details->parentIds.end(), id) != details->parentIds.end(); }); + + for (const std::string& parentId : parentIdsNew) + folderContents_[parentId].childItems.push_back(it); //new insert => no need for duplicate check + + for (const std::string& parentId : parentIdsRemoved) + { + auto itP = folderContents_.find(parentId); + if (itP != folderContents_.end()) + eraseIf(itP->second.childItems, [&](auto itChild) { return itChild == it; }); + } + //if all parents are removed, Google Drive will (recursively) delete the item => don't prematurely do this now: wait for change notifications! + + it->second = *details; + } + else //create + { + auto itNew = itemDetails_.emplace(itemId, *details).first; + + for (const std::string& parentId : details->parentIds) + folderContents_[parentId].childItems.push_back(itNew); //new insert => no need for duplicate check + } + } + else //delete + { + if (it != itemDetails_.end()) + { + for (const std::string& parentId : it->second.parentIds) //1. delete from parent folders + { + auto itP = folderContents_.find(parentId); + if (itP != folderContents_.end()) + eraseIf(itP->second.childItems, [&](auto itChild) { return itChild == it; }); + } + itemDetails_.erase(it); + } + + auto itP = folderContents_.find(itemId); + if (itP != folderContents_.end()) + { + for (auto itChild : itP->second.childItems) //2. delete as parent from child items (don't wait for change notifications of children) + eraseIf(itChild->second.parentIds, [&](const std::string& id) { return id == itemId; }); + folderContents_.erase(itP); + } + } + } + + using DetailsIterator = std::unordered_map<std::string, GoogleItemDetails>::iterator; + + struct FolderContent + { + bool isKnownFolder = false; //=we've seen its full content at least once; further changes are calculated via change notifications! + std::vector<DetailsIterator> childItems; + }; + std::unordered_map<std::string /*folderId*/, FolderContent> folderContents_; + std::unordered_map<std::string /*itemId*/, GoogleItemDetails> itemDetails_; //contains ALL known, existing items! + + std::string lastSyncToken_; //marker corresponding to last sync with Google's change notifications + std::chrono::steady_clock::time_point lastSyncTime_ = std::chrono::steady_clock::now() - GOOGLE_DRIVE_SYNC_INTERVAL; //... with Google Drive (default: sync is due) + + std::vector<std::weak_ptr<ItemIdDelta>> changeLog_; //track changed items since FileStateDelta was created (includes sync with Google + our own intermediate change notifications) + + std::string rootId_; + GoogleAccessBuffer& accessBuf_; +}; + +//========================================================================================== +//========================================================================================== + +class GooglePersistentSessions +{ +public: + GooglePersistentSessions(const Zstring& configDirPath) : configDirPath_(configDirPath) {} + + void saveActiveSessions() //throw FileError + { + std::vector<Protected<SessionHolder>*> protectedSessions; //pointers remain stable, thanks to std::map<> + globalSessions_.access([&](GlobalSessions& sessions) + { + for (auto& [googleUserEmail, protectedSession] : sessions) + protectedSessions.push_back(&protectedSession); + }); + + if (!protectedSessions.empty()) + { + createDirectoryIfMissingRecursion(configDirPath_); //throw FileError + + std::exception_ptr firstError; + + //access each session outside the globalSessions_ lock! + for (Protected<SessionHolder>* protectedSession : protectedSessions) + protectedSession->access([&](SessionHolder& holder) + { + if (holder.session) + try + { + const Zstring dbFilePath = getDbFilePath(holder.session->accessBuf.ref().getUserEmail()); + + //generate (hopefully) unique file name to avoid clashing with unrelated tmp file (concurrent FFS shutdown!) + const Zstring shortGuid = printNumber<Zstring>(Zstr("%04x"), static_cast<unsigned int>(getCrc16(generateGUID()))); + const Zstring dbFilePathTmp = dbFilePath + Zstr('.') + shortGuid + Zstr(".tmp"); + + ZEN_ON_SCOPE_FAIL(try { removeFilePlain(dbFilePathTmp); } + catch (FileError&) {}); + + saveSession(dbFilePathTmp, *holder.session); //throw FileError + + moveAndRenameItem(dbFilePathTmp, dbFilePath, true /*replaceExisting*/); //throw FileError, ErrorDifferentVolume, (ErrorTargetExisting) + } + catch (FileError&) { if (!firstError) firstError = std::current_exception(); } + }); + + if (firstError) + std::rethrow_exception(firstError); //throw FileError + } + } + + Zstring addUserSession(const Zstring& googleLoginHint, const std::function<void()>& updateGui /*throw X*/) //throw FileError, X + { + const GoogleAccessInfo accessInfo = authorizeAccessToGoogleDrive(googleLoginHint, updateGui); //throw FileError, X + + accessUserSession(accessInfo.userInfo.email, [&](std::optional<UserSession>& userSession) //throw FileError + { + if (userSession) + userSession->accessBuf.ref().update(accessInfo); //redundant? + else + { + auto accessBuf = makeSharedRef<GoogleAccessBuffer>(accessInfo); + auto fileState = makeSharedRef<GoogleFileState >(accessBuf.ref()); //throw FileError + userSession = { accessBuf, fileState }; + } + }); + return accessInfo.userInfo.email; + } + + void removeUserSession(const Zstring& googleUserEmail) //throw FileError + { + try + { + accessUserSession(googleUserEmail, [&](std::optional<UserSession>& userSession) //throw FileError + { + if (userSession) + revokeAccessToGoogleDrive(userSession->accessBuf.ref().getAccessToken(), googleUserEmail); //throw FileError + }); + } + catch (FileError&) { assert(false); } //best effort: try to invalidate the access token + //=> expected to fail if offline => not worse than removing FFS via "Uninstall Programs" + + //start with deleting the DB file (1. maybe it's corrupted? 2. skip unnecessary lazy-load) + const Zstring dbFilePath = getDbFilePath(googleUserEmail); + try + { + removeFilePlain(dbFilePath); //throw FileError + } + catch (FileError&) + { + if (itemStillExists(dbFilePath)) //throw FileError + throw; + } + + accessUserSession(googleUserEmail, [&](std::optional<UserSession>& userSession) //throw FileError + { + userSession.reset(); + }); + } + + std::vector<Zstring> /*Google user email*/ listUserSessions() //throw FileError + { + std::vector<Zstring> emails; + + std::vector<Protected<SessionHolder>*> protectedSessions; //pointers remain stable, thanks to std::map<> + globalSessions_.access([&](GlobalSessions& sessions) + { + for (auto& [googleUserEmail, protectedSession] : sessions) + protectedSessions.push_back(&protectedSession); + }); + + //access each session outside the globalSessions_ lock! + for (Protected<SessionHolder>* protectedSession : protectedSessions) + protectedSession->access([&](SessionHolder& holder) + { + if (holder.session) + emails.push_back(holder.session->accessBuf.ref().getUserEmail()); + }); + + //also include available, but not-yet-loaded sessions + traverseFolder(configDirPath_, + [&](const FileInfo& fi) { if (endsWith(fi.itemName, Zstr(".db"))) emails.push_back(beforeLast(fi.itemName, Zstr('.'), IF_MISSING_RETURN_NONE)); }, + [&](const FolderInfo& fi) {}, + [&](const SymlinkInfo& si) {}, + [&](const std::wstring& errorMsg) + { + if (itemStillExists(configDirPath_)) //throw FileError + throw FileError(errorMsg); + }); + + removeDuplicates(emails, LessAsciiNoCase()); + return emails; + } + + struct AsyncAccessInfo + { + std::string accessToken; //don't allow (long-running) web requests while holding the global session lock! + GoogleFileState::FileStateDelta stateDelta; + }; + //perf: amortized fully buffered! + AsyncAccessInfo accessGlobalFileState(const Zstring& googleUserEmail, const std::function<void(GoogleFileState& fileState)>& useFileState /*throw X*/) //throw FileError, X + { + std::string accessToken; + GoogleFileState::FileStateDelta stateDelta; + + accessUserSession(googleUserEmail, [&](std::optional<UserSession>& userSession) //throw FileError + { + if (!userSession) + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), + replaceCpy(_("Please authorize access to user account %x."), L"%x", fmtPath(googleUserEmail))); + + //manage last sync time here rather than in GoogleFileState, so that "lastSyncToken" remains stable while accessing GoogleFileState in the callback + if (userSession->fileState.ref().syncIsDue()) + userSession->fileState.ref().syncWithGoogle(); //throw FileError + + accessToken = userSession->accessBuf.ref().getAccessToken(); //throw FileError + stateDelta = userSession->fileState.ref().registerFileStateDelta(); + + useFileState(userSession->fileState.ref()); //throw X + }); + return { accessToken, stateDelta }; + } + +private: + GooglePersistentSessions (const GooglePersistentSessions&) = delete; + GooglePersistentSessions& operator=(const GooglePersistentSessions&) = delete; + + struct UserSession; + + Zstring getDbFilePath(Zstring googleUserEmail) const + { + for (Zchar& c : googleUserEmail) + c = asciiToLower(c); + //return appendSeparator(configDirPath_) + utfTo<Zstring>(formatAsHexString(getMd5(utfTo<std::string>(googleUserEmail)))) + Zstr(".db"); + return appendSeparator(configDirPath_) + googleUserEmail + Zstr(".db"); + } + + void accessUserSession(const Zstring& googleUserEmail, const std::function<void(std::optional<UserSession>& userSession)>& useSession) //throw FileError + { + Protected<SessionHolder>* protectedSession = nullptr; //pointers remain stable, thanks to std::map<> + globalSessions_.access([&](GlobalSessions& sessions) { protectedSession = &sessions[googleUserEmail]; }); + + protectedSession->access([&](SessionHolder& holder) + { + if (!holder.dbWasLoaded) //let's NOT load the DB files under the globalSessions_ lock, but the session-specific one! + try + { + holder.session = loadSession(getDbFilePath(googleUserEmail)); //throw FileError + } + catch (FileError&) + { + if (itemStillExists(getDbFilePath(googleUserEmail))) //throw FileError + throw; + } + holder.dbWasLoaded = true; + useSession(holder.session); + }); + } + + static void saveSession(const Zstring& dbFilePath, const UserSession& userSession) //throw FileError + { + MemoryStreamOut<ByteArray> streamOut; + + writeArray(streamOut, DB_FORMAT_DESCR, sizeof(DB_FORMAT_DESCR)); + writeNumber<int32_t>(streamOut, DB_FORMAT_VER); + + userSession.accessBuf.ref().serialize(streamOut); + userSession.fileState.ref().serialize(streamOut); + + ByteArray zstreamOut; + try + { + zstreamOut = compress(streamOut.ref(), 3 /*compression level: see db_file.cpp*/); //throw ZlibInternalError + } + catch (ZlibInternalError&) { throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(dbFilePath)), L"zlib internal error"); } + + saveBinContainer(dbFilePath, zstreamOut, nullptr /*notifyUnbufferedIO*/); //throw FileError + } + + static UserSession loadSession(const Zstring& dbFilePath) //throw FileError + { + ByteArray zstream = loadBinContainer<ByteArray>(dbFilePath, nullptr /*notifyUnbufferedIO*/); //throw FileError + ByteArray rawStream; + try + { + rawStream = decompress(zstream); //throw ZlibInternalError + } + catch (ZlibInternalError&) { throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(dbFilePath)), L"Zlib internal error"); } + + MemoryStreamIn<ByteArray> streamIn(rawStream); + try + { + char tmp[sizeof(DB_FORMAT_DESCR)] = {}; + readArray(streamIn, &tmp, sizeof(tmp)); //file format header + const int fileVersion = readNumber<int32_t>(streamIn); // + + if (!std::equal(std::begin(tmp), std::end(tmp), std::begin(DB_FORMAT_DESCR)) || + fileVersion != DB_FORMAT_VER) + throw UnexpectedEndOfStreamError(); //well, not really...!? + + auto accessBuf = makeSharedRef<GoogleAccessBuffer>(streamIn); //throw UnexpectedEndOfStreamError + auto fileState = makeSharedRef<GoogleFileState >(streamIn, accessBuf.ref()); //throw UnexpectedEndOfStreamError + return { accessBuf, fileState }; + } + catch (UnexpectedEndOfStreamError&) + { + throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(dbFilePath)), L"Unexpected end of stream."); + } + } + + struct UserSession + { + SharedRef<GoogleAccessBuffer> accessBuf; + SharedRef<GoogleFileState> fileState; + }; + + struct SessionHolder + { + bool dbWasLoaded = false; + std::optional<UserSession> session; + }; + using GlobalSessions = std::map<Zstring /*Google user email*/, Protected<SessionHolder>, LessAsciiNoCase>; + + Protected<GlobalSessions> globalSessions_; + const Zstring configDirPath_; +}; +//========================================================================================== +Global<GooglePersistentSessions> globalGoogleSessions; +//========================================================================================== + + +GooglePersistentSessions::AsyncAccessInfo accessGlobalFileState(const Zstring& googleUserEmail, const std::function<void(GoogleFileState& fileState)>& useFileState /*throw X*/) //throw FileError, X +{ + const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get(); + if (!gps) + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), + L"Function call not allowed during process init/shutdown."); + + return gps->accessGlobalFileState(googleUserEmail, useFileState); //throw FileError, X +} + +//========================================================================================== +//========================================================================================== + +struct GetDirDetails +{ + GetDirDetails(const GdrivePath& gdriveFolderPath) : gdriveFolderPath_(gdriveFolderPath) {} + + struct Result + { + std::vector<GoogleFileItem> childItems; + GdrivePath gdriveFolderPath; + }; + Result operator()() const + { + try + { + std::string folderId; + std::optional<std::vector<GoogleFileItem>> childItemsBuffered; + const GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(gdriveFolderPath_.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + folderId = fileState.getItemId(gdriveFolderPath_.itemPath); //throw SysError + childItemsBuffered = fileState.tryGetBufferedFolderContent(folderId); + }); + + std::vector<GoogleFileItem> childItems; + if (childItemsBuffered) + childItems = std::move(*childItemsBuffered); + else + { + childItems = readFolderContent(folderId, aai.accessToken, gdriveFolderPath_); //throw FileError + + //buffer new file state ASAP => make sure accessGlobalFileState() has amortized constant access (despite the occasional internal readFolderContent() on non-leaf folders) + accessGlobalFileState(gdriveFolderPath_.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyFolderContent(aai.stateDelta, folderId, childItems); + }); + } + + for (const GoogleFileItem& item : childItems) + if (item.details.itemName.empty()) + throw SysError(L"Folder contains child item without a name."); //mostly an issue for FFS's folder traversal, but NOT for globalGoogleSessions! + + return { std::move(childItems), gdriveFolderPath_ }; + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read directory %x."), L"%x", fmtPath(getGoogleDisplayPath(gdriveFolderPath_))), e.toString()); } + } + +private: + GdrivePath gdriveFolderPath_; +}; + +class SingleFolderTraverser +{ +public: + SingleFolderTraverser(const Zstring& googleUserEmail, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/) : + workload_(workload), googleUserEmail_(googleUserEmail) + { + while (!workload_.empty()) + { + auto wi = std::move(workload_. back()); //yes, no strong exception guarantee (std::bad_alloc) + /**/ workload_.pop_back(); // + const auto& [folderPath, cb] = wi; + + tryReportingDirError([&] //throw X + { + traverseWithException(folderPath, *cb); //throw FileError, X + }, *cb); + } + } + +private: + SingleFolderTraverser (const SingleFolderTraverser&) = delete; + SingleFolderTraverser& operator=(const SingleFolderTraverser&) = delete; + + void traverseWithException(const AfsPath& folderPath, AFS::TraverserCallback& cb) //throw FileError, X + { + const GetDirDetails::Result r = GetDirDetails({ googleUserEmail_, folderPath })(); //throw FileError + + for (const GoogleFileItem& item : r.childItems) + { + const Zstring itemName = utfTo<Zstring>(item.details.itemName); + if (item.details.isFolder) + { + const AfsPath afsItemPath(nativeAppendPaths(r.gdriveFolderPath.itemPath.value, itemName)); + + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ itemName, nullptr /*symlinkInfo*/ })) //throw X + workload_.push_back({ afsItemPath, std::move(cbSub) }); + } + else + { + AFS::FileId fileId = copyStringTo<AFS::FileId>(item.itemId); + cb.onFile({ itemName, item.details.fileSize, item.details.modTime, fileId, nullptr /*symlinkInfo*/ }); //throw X + } + } + } + + std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>> workload_; + const Zstring googleUserEmail_; +}; + + +void gdriveTraverseFolderRecursive(const Zstring& googleUserEmail, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/, size_t) //throw X +{ + SingleFolderTraverser dummy(googleUserEmail, workload); //throw X +} +//========================================================================================== +//========================================================================================== + +struct InputStreamGdrive : public AbstractFileSystem::InputStream +{ + InputStreamGdrive(const GdrivePath& gdrivePath, const IOCallback& notifyUnbufferedIO /*throw X*/) : + gdrivePath_(gdrivePath), + notifyUnbufferedIO_(notifyUnbufferedIO) + { + worker_ = InterruptibleThread([asyncStreamOut = this->asyncStreamIn_, gdrivePath] + { + setCurrentThreadName(("Istream[Gdrive] " + utfTo<std::string>(getGoogleDisplayPath(gdrivePath))). c_str()); + try + { + std::string accessToken; + std::string fileId; + try + { + accessToken = accessGlobalFileState(gdrivePath.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + fileId = fileState.getItemId(gdrivePath.itemPath); //throw SysError + }).accessToken; + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot open file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); } + + auto writeBlock = [&](const void* buffer, size_t bytesToWrite) + { + return asyncStreamOut->write(buffer, bytesToWrite); //throw ThreadInterruption + }; + + gdriveDownloadFile(fileId, writeBlock, accessToken, gdrivePath); //throw FileError, ThreadInterruption + + asyncStreamOut->closeStream(); + } + catch (FileError&) { asyncStreamOut->setWriteError(std::current_exception()); } //let ThreadInterruption pass through! + }); + } + + ~InputStreamGdrive() + { + asyncStreamIn_->setReadError(std::make_exception_ptr(ThreadInterruption())); + worker_.join(); + } + + size_t read(void* buffer, size_t bytesToRead) override //throw FileError, (ErrorFileLocked), X; return "bytesToRead" bytes unless end of stream! + { + const size_t bytesRead = asyncStreamIn_->read(buffer, bytesToRead); //throw FileError + reportBytesProcessed(); //throw X + return bytesRead; + //no need for asyncStreamIn_->checkWriteErrors(): once end of stream is reached, asyncStreamOut->closeStream() was called => no errors occured + } + + size_t getBlockSize() const override { return 64 * 1024; } //non-zero block size is AFS contract! + + std::optional<AFS::StreamAttributes> getAttributesBuffered() override //throw FileError + { + AFS::StreamAttributes attr = {}; + try + { + accessGlobalFileState(gdrivePath_.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + std::pair<std::string /*itemId*/, GoogleItemDetails> gdriveAttr = fileState.getFileAttributes(gdrivePath_.itemPath); //throw SysError + attr.modTime = gdriveAttr.second.modTime; + attr.fileSize = gdriveAttr.second.fileSize; + attr.fileId = copyStringTo<AFS::FileId>(gdriveAttr.first); + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath_))), e.toString()); } + return std::move(attr); //[!] + } + +private: + void reportBytesProcessed() //throw X + { + const int64_t totalBytesDownloaded = asyncStreamIn_->getTotalBytesWritten(); + if (notifyUnbufferedIO_) notifyUnbufferedIO_(totalBytesDownloaded - totalBytesReported_); //throw X + totalBytesReported_ = totalBytesDownloaded; + } + + const GdrivePath gdrivePath_; + const IOCallback notifyUnbufferedIO_; //throw X + int64_t totalBytesReported_ = 0; + std::shared_ptr<AsyncStreamBuffer> asyncStreamIn_ = std::make_shared<AsyncStreamBuffer>(GDRIVE_STREAM_BUFFER_SIZE); + InterruptibleThread worker_; +}; + +//========================================================================================== + +//target existing: 1. fails with "already existing or 2. creates duplicate file! +struct OutputStreamGdrive : public AbstractFileSystem::OutputStreamImpl +{ + OutputStreamGdrive(const GdrivePath& gdrivePath, + std::optional<uint64_t> streamSize, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) : + gdrivePath_(gdrivePath), + notifyUnbufferedIO_(notifyUnbufferedIO) + { + std::promise<AFS::FileId> pFileId; + futFileId_ = pFileId.get_future(); + + //PathAccessLock? Not needed, because the AFS abstraction allows for "undefined behavior" + + worker_ = InterruptibleThread([asyncStreamIn = this->asyncStreamOut_, gdrivePath, streamSize, modTime, pFileId = std::move(pFileId)]() mutable + { + setCurrentThreadName(("Ostream[Gdrive] " + utfTo<std::string>(getGoogleDisplayPath(gdrivePath))). c_str()); + try + { + try + { + GoogleFileState::PathStatus ps; + GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(gdrivePath.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + ps = fileState.getPathStatus(gdrivePath.itemPath); //throw SysError + }); + if (ps.relPath.empty()) + throw SysError(formatSystemErrorRaw(EEXIST)); + if (ps.relPath.size() > 1) //parent folder missing + throw SysError(replaceCpy(_("Cannot find %x."), L"%x", + fmtPath(getGoogleDisplayPath({ gdrivePath.userEmail, AfsPath(nativeAppendPaths(ps.existingPath.value, ps.relPath.front()))})))); + + const Zstring fileName = AFS::getItemName(gdrivePath.itemPath); + const std::string& parentFolderId = ps.existingItemId; + + auto readBlock = [&](void* buffer, size_t bytesToRead) + { + //returns "bytesToRead" bytes unless end of stream! => maps nicely into Posix read() semantics expected by gdriveUploadFile() + return asyncStreamIn->read(buffer, bytesToRead); //throw ThreadInterruption + }; + + //for whatever reason, gdriveUploadFile() is equally-fast or faster than gdriveUploadSmallFile(), despite its two roundtrips, even when the file sizes are 0!! + //=> issue likely on Google's side + const std::string fileIdNew = //streamSize && *streamSize < 5 * 1024 * 1024 ? + //gdriveUploadSmallFile(fileName, parentFolderId, *streamSize, modTime, readBlock, aai.accessToken, gdrivePath) : //throw FileError, ThreadInterruption + gdriveUploadFile (fileName, parentFolderId, modTime, readBlock, aai.accessToken, gdrivePath); //throw FileError, ThreadInterruption + assert(asyncStreamIn->getTotalBytesRead() == asyncStreamIn->getTotalBytesWritten()); + (void)streamSize; + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + GoogleFileItem newFileItem = {}; + newFileItem.itemId = fileIdNew; + newFileItem.details.itemName = utfTo<std::string>(fileName); + newFileItem.details.isFolder = false; + newFileItem.details.fileSize = asyncStreamIn->getTotalBytesRead(); + if (modTime) //else: whatever modTime Google Drive selects will be notified after GOOGLE_DRIVE_SYNC_INTERVAL + newFileItem.details.modTime = *modTime; + newFileItem.details.parentIds.push_back(parentFolderId); + + accessGlobalFileState(gdrivePath.userEmail, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyItemCreated(aai.stateDelta, newFileItem); + }); + + pFileId.set_value(copyStringTo<AFS::FileId>(fileIdNew)); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(getGoogleDisplayPath(gdrivePath))), e.toString()); } + } + catch (FileError&) { asyncStreamIn->setReadError(std::current_exception()); } //let ThreadInterruption pass through! + }); + } + + ~OutputStreamGdrive() + { + if (worker_.joinable()) + { + asyncStreamOut_->setWriteError(std::make_exception_ptr(ThreadInterruption())); + worker_.join(); + } + } + + void write(const void* buffer, size_t bytesToWrite) override //throw FileError, X + { + asyncStreamOut_->write(buffer, bytesToWrite); //throw FileError + reportBytesProcessed(); //throw X + } + + AFS::FinalizeResult finalize() override //throw FileError, X + { + asyncStreamOut_->closeStream(); + + while (!worker_.tryJoinFor(std::chrono::milliseconds(50))) + reportBytesProcessed(); //throw X + reportBytesProcessed(); //[!] once more, now that *all* bytes were written + + asyncStreamOut_->checkReadErrors(); //throw FileError + //-------------------------------------------------------------------- + AFS::FinalizeResult result; + assert(isReady(futFileId_)); //*must* be available since file creation completed successfully at this point + result.fileId = futFileId_.get(); + //result.errorModTime -> already (successfully) set during file creation + return result; + } + +private: + void reportBytesProcessed() //throw X + { + const int64_t totalBytesUploaded = asyncStreamOut_->getTotalBytesRead(); + if (notifyUnbufferedIO_) notifyUnbufferedIO_(totalBytesUploaded - totalBytesReported_); //throw X + totalBytesReported_ = totalBytesUploaded; + } + + const GdrivePath gdrivePath_; + const IOCallback notifyUnbufferedIO_; //throw X + int64_t totalBytesReported_ = 0; + std::shared_ptr<AsyncStreamBuffer> asyncStreamOut_ = std::make_shared<AsyncStreamBuffer>(GDRIVE_STREAM_BUFFER_SIZE); + InterruptibleThread worker_; + std::future<AFS::FileId> futFileId_; //"play it safe", however with our current access pattern, also could have used an unprotected AFS::FileId +}; + +//========================================================================================== + +class GdriveFileSystem : public AbstractFileSystem +{ +public: + GdriveFileSystem(const Zstring& googleUserEmail) : googleUserEmail_(googleUserEmail) {} + +private: + GdrivePath getGdrivePath(const AfsPath& afsPath) const { return { googleUserEmail_, afsPath }; } + + Zstring getInitPathPhrase(const AfsPath& afsPath) const override { return concatenateGoogleFolderPathPhrase(getGdrivePath(afsPath)); } + + std::wstring getDisplayPath(const AfsPath& afsPath) const override { return getGoogleDisplayPath(getGdrivePath(afsPath)); } + + bool isNullFileSystem() const override { return googleUserEmail_.empty(); } + + int compareDeviceSameAfsType(const AbstractFileSystem& afsRhs) const override + { + return compareAsciiNoCase(googleUserEmail_, static_cast<const GdriveFileSystem&>(afsRhs).googleUserEmail_); + } + + //---------------------------------------------------------------------------------------------------------------- + ItemType getItemType(const AfsPath& afsPath) const override //throw FileError + { + if (std::optional<ItemType> type = itemStillExists(afsPath)) //throw FileError + return *type; + throw FileError(replaceCpy(_("Cannot find %x."), L"%x", fmtPath(getDisplayPath(afsPath)))); + } + + std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const override //throw FileError + { + try + { + GoogleFileState::PathStatus ps; + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + ps = fileState.getPathStatus(afsPath); //throw SysError + }); + if (ps.relPath.empty()) + return ps.existingIsFolder ? ItemType::FOLDER : ItemType::FILE; + return {}; + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + //---------------------------------------------------------------------------------------------------------------- + + //already existing: fail/ignore + //=> we choose to let Google Drive fail and give a clear error message + void createFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + //avoid duplicate Google Drive item creation by multiple threads + PathAccessLock pal(getGdrivePath(afsPath)); //throw SysError + + GoogleFileState::PathStatus ps; + const GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + ps = fileState.getPathStatus(afsPath); //throw SysError + }); + + if (ps.relPath.empty()) + throw SysError(formatSystemErrorRaw(EEXIST)); + if (ps.relPath.size() > 1) //parent folder missing + throw SysError(replaceCpy(_("Cannot find %x."), L"%x", fmtPath(getDisplayPath(AfsPath(nativeAppendPaths(ps.existingPath.value, ps.relPath.front())))))); + + const Zstring folderName = getItemName(afsPath); + const std::string& parentFolderId = ps.existingItemId; + + const std::string folderIdNew = gdriveCreateFolderPlain(folderName, parentFolderId, aai.accessToken); //throw FileError, SysError + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyFolderCreated(aai.stateDelta, folderIdNew, folderName, parentFolderId); + }); + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot create directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeItemPlainImpl(const AfsPath& afsPath) const //throw FileError, SysError + { + std::string itemId; + std::optional<std::string> parentIdToUnlink; + const GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + const std::optional<AfsPath> parentPath = getParentPath(afsPath); + if (!parentPath) throw SysError(L"Item is device root"); + + std::pair<std::string /*itemId*/, GoogleItemDetails> gdriveAttr = fileState.getFileAttributes(afsPath); //throw SysError + itemId = gdriveAttr.first; + assert(gdriveAttr.second.parentIds.size() > 1 || + (gdriveAttr.second.parentIds.size() == 1 && gdriveAttr.second.parentIds[0] == fileState.getItemId(*parentPath))); + + if (gdriveAttr.second.parentIds.size() != 1) //hard-link handling + parentIdToUnlink = fileState.getItemId(*parentPath); //throw SysError + }); + + if (parentIdToUnlink) + { + gdriveUnlinkParent(itemId, *parentIdToUnlink, aai.accessToken); //throw FileError, SysError + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyParentRemoved(aai.stateDelta, itemId, *parentIdToUnlink); + }); + } + else + { + gdriveDeleteItem(itemId, aai.accessToken); //throw FileError, SysError + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyItemDeleted(aai.stateDelta, itemId); + }); + } + } + + void removeFilePlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + removeItemPlainImpl(afsPath); //throw FileError, SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot delete file %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + + void removeSymlinkPlain(const AfsPath& afsPath) const override //throw FileError + { + throw FileError(replaceCpy(_("Cannot delete symbolic link %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Symlinks not supported for Google Drive."); + } + + void removeFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + removeItemPlainImpl(afsPath); //throw FileError, SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot delete directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + + void removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const override //one call for each object! + { + if (onBeforeFolderDeletion) onBeforeFolderDeletion(getDisplayPath(afsPath)); //throw X + try + { + //deletes recursively with a single call! + removeFolderPlain(afsPath); //throw FileError + } + catch (const FileError&) + { + if (!itemStillExists(afsPath)) //throw FileError + return; + throw; + } + } + + //---------------------------------------------------------------------------------------------------------------- + AbstractPath getSymlinkResolvedPath(const AfsPath& afsPath) const override //throw FileError + { + throw FileError(replaceCpy(_("Cannot determine final path for %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Symlinks not supported for Google Drive."); + } + + std::string getSymlinkBinaryContent(const AfsPath& afsPath) const override //throw FileError + { + throw FileError(replaceCpy(_("Cannot resolve symbolic link %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Symlinks not supported for Google Drive."); + } + //---------------------------------------------------------------------------------------------------------------- + + //return value always bound: + std::unique_ptr<InputStream> getInputStream(const AfsPath& afsPath, const IOCallback& notifyUnbufferedIO /*throw X*/) const override //throw FileError, (ErrorFileLocked) + { + return std::make_unique<InputStreamGdrive>(getGdrivePath(afsPath), notifyUnbufferedIO); + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + //=> actual behavior: 1. fails with "already existing or 2. creates duplicate file! + //=> we choose to let Google Drive create a duplicate file, because setting PathAccessLock for a potentially long-running write operation is excessive! + std::unique_ptr<OutputStreamImpl> getOutputStream(const AfsPath& afsPath, //throw FileError + std::optional<uint64_t> streamSize, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + //target existing: 1. fails with "already existing or 2. creates duplicate file! + return std::make_unique<OutputStreamGdrive>(getGdrivePath(afsPath), streamSize, modTime, notifyUnbufferedIO); + } + + //---------------------------------------------------------------------------------------------------------------- + void traverseFolderRecursive(const TraverserWorkload& workload /*throw X*/, size_t parallelOps) const override + { + gdriveTraverseFolderRecursive(googleUserEmail_, workload, parallelOps); //throw X + } + //---------------------------------------------------------------------------------------------------------------- + + //symlink handling: follow link! + //target existing: undefined behavior! (fail/overwrite/auto-rename) + FileCopyResult copyFileForSameAfsType(const AfsPath& afsPathSource, const StreamAttributes& attrSource, //throw FileError, (ErrorFileLocked), X + const AbstractPath& apTarget, bool copyFilePermissions, const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + //no native Google Drive file copy => use stream-based file copy: + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for Google Drive."); + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + return copyFileAsStream(afsPathSource, attrSource, apTarget, notifyUnbufferedIO); //throw FileError, (ErrorFileLocked), X + } + + //already existing: fail/ignore + //symlink handling: follow link! + void copyNewFolderForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override //throw FileError + { + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for Google Drive."); + + //already existing: fail/ignore + AFS::createFolderPlain(apTarget); //throw FileError + } + + void copySymlinkForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override //throw FileError + { + throw FileError(replaceCpy(replaceCpy(_("Cannot copy symbolic link %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))), + L"Symlinks not supported for Google Drive."); + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + //=> actual behavior: fails with "already existing + void moveAndRenameItemForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget) const override //throw FileError, ErrorDifferentVolume + { + auto generateErrorMsg = [&] { return replaceCpy(replaceCpy(_("Cannot move file %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))); + }; + + if (compareDeviceSameAfsType(apTarget.afsDevice.ref()) != 0) + throw ErrorDifferentVolume(generateErrorMsg(), L"Different Google Drive volume."); + + try + { + //avoid duplicate Google Drive item creation by multiple threads + PathAccessLock pal(getGdrivePath(apTarget.afsPath)); //throw SysError + + const Zstring itemNameOld = getItemName(afsPathSource); + const Zstring itemNameNew = AFS::getItemName(apTarget); + + const std::optional<AfsPath> parentAfsPathSource = getParentPath(afsPathSource); + const std::optional<AfsPath> parentAfsPathTarget = getParentPath(apTarget.afsPath); + if (!parentAfsPathSource) throw SysError(L"Source is device root"); + if (!parentAfsPathTarget) throw SysError(L"Target is device root"); + + std::string itemIdSource; + time_t modTimeSource = 0; + std::string parentIdSource; + std::string parentIdTarget; + const GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + std::pair<std::string /*itemId*/, GoogleItemDetails> gdriveAttr = fileState.getFileAttributes(afsPathSource); //throw SysError + itemIdSource = gdriveAttr.first; + modTimeSource = gdriveAttr.second.modTime; + parentIdSource = fileState.getItemId(*parentAfsPathSource); //throw SysError + GoogleFileState::PathStatus psTarget = fileState.getPathStatus(apTarget.afsPath); //throw SysError + + //e.g. changing file name case only => this is not an "already exists" situation! + //also: hardlink referenced by two different paths, the source one will be unlinked + if (psTarget.relPath.empty() && psTarget.existingItemId == itemIdSource) + parentIdTarget = fileState.getItemId(*parentAfsPathTarget); //throw SysError + else + { + if (psTarget.relPath.empty()) + throw SysError(formatSystemErrorRaw(EEXIST)); + if (psTarget.relPath.size() > 1) //parent folder missing + throw SysError(replaceCpy(_("Cannot find %x."), L"%x", + fmtPath(getDisplayPath(AfsPath(nativeAppendPaths(psTarget.existingPath.value, psTarget.relPath.front())))))); + parentIdTarget = psTarget.existingItemId; + } + }); + + if (parentIdSource == parentIdTarget && itemNameOld == itemNameNew) + return; //nothing to do + + //target name already existing? will (happily) create duplicate items + gdriveMoveAndRenameItem(itemIdSource, parentIdSource, parentIdTarget, itemNameNew, modTimeSource, aai.accessToken); //throw FileError, SysError + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + fileState.notifyMoveAndRename(aai.stateDelta, itemIdSource, parentIdSource, parentIdTarget, itemNameNew); + }); + } + catch (const SysError& e) { throw FileError(generateErrorMsg(), e.toString()); } + } + + bool supportsPermissions(const AfsPath& afsPath) const override { return false; } //throw FileError + + //---------------------------------------------------------------------------------------------------------------- + ImageHolder getFileIcon (const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } //noexcept; optional return value + ImageHolder getThumbnailImage(const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } //noexcept; optional return value + + void authenticateAccess(bool allowUserInteraction) const override //throw FileError + { + if (allowUserInteraction) + try + { + const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get(); + if (!gps) + throw SysError(L"Function call not allowed during process init/shutdown."); + + for (const Zstring& email : gps->listUserSessions()) //throw FileError + if (equalAsciiNoCase(email, googleUserEmail_)) + return; + gps->addUserSession(googleUserEmail_ /*googleLoginHint*/, nullptr /*updateGui*/); //throw FileError + //error messages will be lost after time out in dir_exist_async.h! However: + //The most-likely-to-fail parts (web access) are reported by authorizeAccessToGoogleDrive() via the browser! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(getDisplayPath(AfsPath()))), e.toString()); } + } + + int getAccessTimeout() const override { return static_cast<int>(std::chrono::seconds(HTTP_SESSION_ACCESS_TIME_OUT).count()); } //returns "0" if no timeout in force + + bool hasNativeTransactionalCopy() const override { return true; } + //---------------------------------------------------------------------------------------------------------------- + + uint64_t getFreeDiskSpace(const AfsPath& afsPath) const override //throw FileError, returns 0 if not available + { + try + { + const std::string& accessToken = accessGlobalFileState(googleUserEmail_, [](GoogleFileState& fileState) {}).accessToken; //throw FileError + return gdriveGetFreeDiskSpace(accessToken); //throw FileError, SysError; returns 0 if not available + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot determine free disk space for %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + + bool supportsRecycleBin(const AfsPath& afsPath, const std::function<void ()>& onUpdateGui) const override { return true; } //throw FileError + + struct RecycleSessionGdrive : public RecycleSession + { + void recycleItemIfExists(const AbstractPath& itemPath, const Zstring& logicalRelPath) override { AFS::recycleItemIfExists(itemPath); } //throw FileError + void tryCleanup(const std::function<void (const std::wstring& displayPath)>& notifyDeletionStatus) override {}; //throw FileError + }; + std::unique_ptr<RecycleSession> createRecyclerSession(const AfsPath& afsPath) const override //throw FileError, return value must be bound! + { + return std::make_unique<RecycleSessionGdrive>(); + } + + void recycleItemIfExists(const AfsPath& afsPath) const override //throw FileError + { + try + { + GoogleFileState::PathStatus ps; + const GooglePersistentSessions::AsyncAccessInfo aai = accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + ps = fileState.getPathStatus(afsPath); //throw SysError + }); + if (ps.relPath.empty()) + { + gdriveMoveToTrash(ps.existingItemId, aai.accessToken); //throw FileError, SysError + + //buffer new file state ASAP (don't wait GOOGLE_DRIVE_SYNC_INTERVAL) + accessGlobalFileState(googleUserEmail_, [&](GoogleFileState& fileState) //throw FileError + { + //a hardlink with multiple parents will be not be accessible anymore via any of its path aliases! + fileState.notifyItemDeleted(aai.stateDelta, ps.existingItemId); + }); + } + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to move %x to the recycle bin."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + + const Zstring googleUserEmail_; +}; +//=========================================================================================================================== +} + + +void fff::googleDriveInit(const Zstring& configDirPath, const Zstring& caCertFilePath) +{ + assert(!httpSessionManager.get()); + httpSessionManager.set(std::make_unique<HttpSessionManager>(caCertFilePath)); + + assert(!globalGoogleSessions.get()); + globalGoogleSessions.set(std::make_unique<GooglePersistentSessions>(configDirPath)); +} + + +void fff::googleDriveTeardown() +{ + try //don't use ~GooglePersistentSessions() to save! Might never happen, e.g. detached thread waiting for Google Drive authentication; terminated on exit! + { + if (const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get()) + gps->saveActiveSessions(); //throw FileError + } + catch (FileError&) { assert(false); } + + assert(globalGoogleSessions.get()); + globalGoogleSessions.set(nullptr); + + assert(httpSessionManager.get()); + httpSessionManager.set(nullptr); +} + + +Zstring fff::googleAddUser(const std::function<void()>& updateGui /*throw X*/) //throw FileError, X +{ + if (const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get()) + return gps->addUserSession(Zstr("") /*googleLoginHint*/, updateGui); //throw FileError, X + + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", L"Google Drive"), L"Function call not allowed during process init/shutdown."); +} + + +void fff::googleRemoveUser(const Zstring& googleUserEmail) //throw FileError +{ + if (const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get()) + return gps->removeUserSession(googleUserEmail); //throw FileError + + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(getGoogleDisplayPath({ googleUserEmail, AfsPath() }))), + L"Function call not allowed during process init/shutdown."); +} + + +std::vector<Zstring> /*Google user email*/ fff::googleListConnectedUsers() //throw FileError +{ + if (const std::shared_ptr<GooglePersistentSessions> gps = globalGoogleSessions.get()) + return gps->listUserSessions(); //throw FileError + + throw FileError(replaceCpy(_("Unable to access %x."), L"%x", L"Google Drive"), L"Function call not allowed during process init/shutdown."); +} + + +Zstring fff::condenseToGoogleFolderPathPhrase(const Zstring& userEmail, const Zstring& relPath) //noexcept +{ + return concatenateGoogleFolderPathPhrase({ trimCpy(userEmail), sanitizeRootRelativePath(relPath) }); +} + + +//e.g.: gdrive:/zenju@gmx.net/folder/file.txt +GdrivePath fff::getResolvedGooglePath(const Zstring& folderPathPhrase) //noexcept +{ + Zstring path = folderPathPhrase; + path = expandMacros(path); //expand before trimming! + trim(path); + + if (startsWithAsciiNoCase(path, googleDrivePrefix)) + path = path.c_str() + strLength(googleDrivePrefix); + + const AfsPath sanPath = sanitizeRootRelativePath(path); //Win/macOS compatibility: let's ignore slash/backslash differences + + const Zstring userEmail = beforeFirst(sanPath.value, FILE_NAME_SEPARATOR, IF_MISSING_RETURN_ALL); + const AfsPath afsPath (afterFirst(sanPath.value, FILE_NAME_SEPARATOR, IF_MISSING_RETURN_NONE)); + + return { userEmail, afsPath }; +} + + +bool fff::acceptsItemPathPhraseGdrive(const Zstring& itemPathPhrase) //noexcept +{ + Zstring path = expandMacros(itemPathPhrase); //expand before trimming! + trim(path); + return startsWithAsciiNoCase(path, googleDrivePrefix); +} + + +AbstractPath fff::createItemPathGdrive(const Zstring& itemPathPhrase) //noexcept +{ + const GdrivePath gdrivePath = getResolvedGooglePath(itemPathPhrase); //noexcept + return AbstractPath(makeSharedRef<GdriveFileSystem>(gdrivePath.userEmail), gdrivePath.itemPath); +} diff --git a/FreeFileSync/Source/fs/gdrive.h b/FreeFileSync/Source/fs/gdrive.h new file mode 100644 index 00000000..d81620a3 --- /dev/null +++ b/FreeFileSync/Source/fs/gdrive.h @@ -0,0 +1,39 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef FS_GDRIVE_9238425018342701356 +#define FS_GDRIVE_9238425018342701356 + +#include "abstract.h" + +namespace fff +{ +bool acceptsItemPathPhraseGdrive(const Zstring& itemPathPhrase); //noexcept +AbstractPath createItemPathGdrive(const Zstring& itemPathPhrase); //noexcept + +//------------------------------------------------------- + +void googleDriveInit(const Zstring& configDirPath, //directory to store Google-Drive-specific files + const Zstring& caCertFilePath); //cacert.pem +void googleDriveTeardown(); + +Zstring /*Google user email*/ googleAddUser(const std::function<void()>& updateGui /*throw X*/); //throw FileError, X +void googleRemoveUser(const Zstring& googleUserEmail); //throw FileError +std::vector<Zstring> /*Google user email*/ googleListConnectedUsers(); //throw FileError + + +struct GdrivePath +{ + Zstring userEmail; + AfsPath itemPath; //path relative to Google Drive root => no leading or trailing backslash! +}; +GdrivePath getResolvedGooglePath(const Zstring& folderPathPhrase); //noexcept + +//expects (potentially messy) user input: +Zstring condenseToGoogleFolderPathPhrase(const Zstring& userEmail, const Zstring& relPath); //noexcept +} + +#endif //FS_GDRIVE_9238425018342701356 diff --git a/FreeFileSync/Source/fs/init_curl_libssh2.cpp b/FreeFileSync/Source/fs/init_curl_libssh2.cpp new file mode 100644 index 00000000..4f4dfbbe --- /dev/null +++ b/FreeFileSync/Source/fs/init_curl_libssh2.cpp @@ -0,0 +1,166 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "init_curl_libssh2.h" +#include <cassert> +#include <zen/thread.h> +#include <zen/file_error.h> +#include "libssh2/init_open_ssl.h" +#include "libssh2/init_libssh2.h" +#include "libcurl/curl_wrap.h" //DON'T include <curl/curl.h> directly! + + +using namespace zen; + + +namespace +{ +int uniInitLevel = 0; //support interleaving initialization calls! (e.g. use for libssh2 and libCurl) +//zero-initialized POD => not subject to static initialization order fiasco + +void libsshCurlUnifiedInit() +{ + assert(runningMainThread()); //all OpenSSL/libssh2/libcurl require init on main thread! + assert(uniInitLevel >= 0); + if (++uniInitLevel != 1) //non-atomic => also require call from main thread + return; + + + openSslInit(); + libssh2Init(); + + const CURLcode rc2 = ::curl_global_init(CURL_GLOBAL_NOTHING /*CURL_GLOBAL_DEFAULT = CURL_GLOBAL_SSL|CURL_GLOBAL_WIN32*/); + assert(rc2 == CURLE_OK); + (void)rc2; +} + + +void libsshCurlUnifiedTearDown() +{ + assert(runningMainThread()); //symmetry with libsshCurlUnifiedInit + assert(uniInitLevel >= 1); + if (--uniInitLevel != 0) + return; + + ::curl_global_cleanup(); + libssh2TearDown(); + openSslTearDown(); + +} +} + + +class zen::UniSessionCounter::Impl +{ +public: + void inc() //throw SysError + { + { + std::unique_lock dummy(lockCount_); + assert(sessionCount_ >= 0); + + if (!newSessionsAllowed_) + throw SysError(L"Function call not allowed during process shutdown."); + + ++sessionCount_; + } + conditionCountChanged_.notify_all(); + } + + void dec() //noexcept + { + { + std::unique_lock dummy(lockCount_); + assert(sessionCount_ >= 1); + --sessionCount_; + } + conditionCountChanged_.notify_all(); + } + + void onInitCompleted() //noexcept + { + std::unique_lock dummy(lockCount_); + newSessionsAllowed_ = true; + } + + void onBeforeTearDown() //noexcept + { + std::unique_lock dummy(lockCount_); + newSessionsAllowed_ = false; + conditionCountChanged_.wait(dummy, [this] { return sessionCount_ == 0; }); + } + + Impl() {} + ~Impl() + { + } + +private: + Impl (const Impl&) = delete; + Impl& operator=(const Impl&) = delete; + + std::mutex lockCount_; + int sessionCount_ = 0; + std::condition_variable conditionCountChanged_; + + bool newSessionsAllowed_ = false; +}; + + +UniSessionCounter::UniSessionCounter() : pimpl(std::make_unique<Impl>()) {} +UniSessionCounter::~UniSessionCounter() {} + + +std::unique_ptr<UniSessionCounter> zen::createUniSessionCounter() +{ + return std::make_unique<UniSessionCounter>(); +} + + +class zen::UniCounterCookie +{ +public: + UniCounterCookie(const std::shared_ptr<UniSessionCounter>& sessionCounter) : sessionCounter_(sessionCounter) {} + ~UniCounterCookie() { sessionCounter_->pimpl->dec(); } + +private: + UniCounterCookie (const UniCounterCookie&) = delete; + UniCounterCookie& operator=(const UniCounterCookie&) = delete; + + const std::shared_ptr<UniSessionCounter> sessionCounter_; +}; + + +std::shared_ptr<UniCounterCookie> zen::getLibsshCurlUnifiedInitCookie(Global<UniSessionCounter>& globalSftpSessionCount) //throw SysError +{ + std::shared_ptr<UniSessionCounter> sessionCounter = globalSftpSessionCount.get(); + if (!sessionCounter) + throw SysError(L"Function call not allowed during process startup/shutdown."); //=> ~UniCounterCookie() *not* called! + sessionCounter->pimpl->inc(); //throw SysError // + + //pass "ownership" of having to call UniSessionCounter::dec() + return std::make_shared<UniCounterCookie>(sessionCounter); //throw SysError +} + + +UniInitializer::UniInitializer(UniSessionCounter& sessionCount) : sessionCount_(sessionCount) +{ + libsshCurlUnifiedInit(); + sessionCount_.pimpl->onInitCompleted(); +} + + +UniInitializer::~UniInitializer() +{ + //wait until all (S)FTP sessions running on detached threads have ended! otherwise they'll crash during ::WSACleanup()! + sessionCount_.pimpl->onBeforeTearDown(); + /* + alternatively we could use a Global<UniInitializer> and have each session own a shared_ptr<UniInitializer>: + drawback 1: SFTP clean-up may happen on worker thread => probably not supported!!! + drawback 2: cleanup will not happen when the C++ runtime on Windows kills all worker threads during shutdown + */ + libsshCurlUnifiedTearDown(); +} diff --git a/FreeFileSync/Source/fs/init_curl_libssh2.h b/FreeFileSync/Source/fs/init_curl_libssh2.h new file mode 100644 index 00000000..57204e78 --- /dev/null +++ b/FreeFileSync/Source/fs/init_curl_libssh2.h @@ -0,0 +1,51 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef INIT_CURL_LIBSSH2_H_4570285702375915765 +#define INIT_CURL_LIBSSH2_H_4570285702375915765 + +#include <memory> +#include <zen/globals.h> + + +namespace zen +{ +//(S)FTP initialization/shutdown dance: + +//1. create "Global<UniSessionCounter> globalSftpSessionCount(createUniSessionCounter());" to have a waitable counter of existing (S)FTP sessions +struct UniSessionCounter +{ + UniSessionCounter(); + ~UniSessionCounter(); + + class Impl; + const std::unique_ptr<Impl> pimpl; +}; +std::unique_ptr<UniSessionCounter> createUniSessionCounter(); + + +//2. count number of existing (S)FTP sessions => tie to (S)FTP session instances! +class UniCounterCookie; +std::shared_ptr<UniCounterCookie> getLibsshCurlUnifiedInitCookie(Global<UniSessionCounter>& globalSftpSessionCount); //throw SysError + + +//3. Create static "UniInitializer globalStartupInitSftp(*globalSftpSessionCount.get());" instance *before* constructing objects like "SftpSessionManager" +// => ~SftpSessionManager will run first and all remaining sessions are on non-main threads => can be waited on in ~UniInitializer +class UniInitializer +{ +public: + UniInitializer(UniSessionCounter& sessionCount); + ~UniInitializer(); + +private: + UniInitializer (const UniInitializer&) = delete; + UniInitializer& operator=(const UniInitializer&) = delete; + + UniSessionCounter& sessionCount_; +}; +} + +#endif //INIT_CURL_LIBSSH2_H_4570285702375915765 diff --git a/FreeFileSync/Source/fs/libcurl/curl_wrap.h b/FreeFileSync/Source/fs/libcurl/curl_wrap.h new file mode 100644 index 00000000..7a5a4f45 --- /dev/null +++ b/FreeFileSync/Source/fs/libcurl/curl_wrap.h @@ -0,0 +1,130 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef CURL_WRAP_H_2879058325032785032789645 +#define CURL_WRAP_H_2879058325032785032789645 + +#include <zen/scope_guard.h> + + + +//------------------------------------------------- +#include <curl/curl.h> +//------------------------------------------------- + + +namespace zen +{ +namespace +{ +std::wstring formatCurlErrorRaw(CURLcode ec) +{ + switch (ec) + { + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OK); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_UNSUPPORTED_PROTOCOL); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FAILED_INIT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_URL_MALFORMAT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_NOT_BUILT_IN); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_COULDNT_RESOLVE_PROXY); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_COULDNT_RESOLVE_HOST); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_COULDNT_CONNECT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_WEIRD_SERVER_REPLY); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_REMOTE_ACCESS_DENIED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_ACCEPT_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_WEIRD_PASS_REPLY); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_ACCEPT_TIMEOUT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_WEIRD_PASV_REPLY); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_WEIRD_227_FORMAT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_CANT_GET_HOST); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_HTTP2); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_COULDNT_SET_TYPE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_PARTIAL_FILE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_COULDNT_RETR_FILE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE20); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_QUOTE_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_HTTP_RETURNED_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_WRITE_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE24); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_UPLOAD_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_READ_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OUT_OF_MEMORY); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OPERATION_TIMEDOUT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE29); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_PORT_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_COULDNT_USE_REST); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE32); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_RANGE_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_HTTP_POST_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_CONNECT_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_BAD_DOWNLOAD_RESUME); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FILE_COULDNT_READ_FILE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_LDAP_CANNOT_BIND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_LDAP_SEARCH_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE40); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FUNCTION_NOT_FOUND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_ABORTED_BY_CALLBACK); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_BAD_FUNCTION_ARGUMENT); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE44); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_INTERFACE_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE46); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TOO_MANY_REDIRECTS); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_UNKNOWN_OPTION); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TELNET_OPTION_SYNTAX); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE50); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE51); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_PEER_FAILED_VERIFICATION); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_GOT_NOTHING); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_ENGINE_NOTFOUND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_ENGINE_SETFAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SEND_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_RECV_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_OBSOLETE57); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_CERTPROBLEM); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_CIPHER); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_BAD_CONTENT_ENCODING); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_LDAP_INVALID_URL); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FILESIZE_EXCEEDED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_USE_SSL_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SEND_FAIL_REWIND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_ENGINE_INITFAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_LOGIN_DENIED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TFTP_NOTFOUND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TFTP_PERM); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_REMOTE_DISK_FULL); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TFTP_ILLEGAL); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TFTP_UNKNOWNID); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_REMOTE_FILE_EXISTS); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_TFTP_NOSUCHUSER); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_CONV_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_CONV_REQD); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_CACERT_BADFILE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_REMOTE_FILE_NOT_FOUND); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSH); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_SHUTDOWN_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_AGAIN); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_CRL_BADFILE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_ISSUER_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_PRET_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_RTSP_CSEQ_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_RTSP_SESSION_ERROR); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_FTP_BAD_FILE_LIST); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_CHUNK_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_NO_CONNECTION_AVAILABLE); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_PINNEDPUBKEYNOTMATCH); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_SSL_INVALIDCERTSTATUS); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_HTTP2_STREAM); + ZEN_CHECK_CASE_FOR_CONSTANT(CURLE_RECURSIVE_API_CALL); + ZEN_CHECK_CASE_FOR_CONSTANT(CURL_LAST); + } + return L"Unknown Curl error: " + numberTo<std::wstring>(ec); +} +} +} + +#else +#error Why is this header already defined? Do not include in other headers: encapsulate the gory details! +#endif //CURL_WRAP_H_2879058325032785032789645 diff --git a/FreeFileSync/Source/fs/libssh2/init_libssh2.cpp b/FreeFileSync/Source/fs/libssh2/init_libssh2.cpp new file mode 100644 index 00000000..b793357d --- /dev/null +++ b/FreeFileSync/Source/fs/libssh2/init_libssh2.cpp @@ -0,0 +1,37 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "init_libssh2.h" +#include <cassert> +#include <libssh2_sftp.h> +#include <openssl/opensslv.h> + + +#ifndef LIBSSH2_OPENSSL + #error check code when/if-ever the OpenSSL libssh2 backend is changed +#endif + +#if OPENSSL_VERSION_NUMBER < 0x10100000L + #error OpenSSL version too old +#endif + + +void zen::libssh2Init() +{ + const int rc = ::libssh2_init(0); + //we need libssh2's crypto init: + // - initializes a few statically allocated constants => avoid (minor) race condition if these were initialized by worker threads + // - enable proper clean up of these variables in libssh2_exit() (otherwise: memory leaks!) + // - there are a few other OpenSSL-related initializations which might be needed (and hopefully won't hurt...) + assert(rc == 0); //libssh2 unconditionally returns 0 => why then have a return value in first place??? + (void)rc; +} + + +void zen::libssh2TearDown() +{ + ::libssh2_exit(); +} diff --git a/FreeFileSync/Source/fs/libssh2/init_libssh2.h b/FreeFileSync/Source/fs/libssh2/init_libssh2.h new file mode 100644 index 00000000..a159850a --- /dev/null +++ b/FreeFileSync/Source/fs/libssh2/init_libssh2.h @@ -0,0 +1,16 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef INIT_LIBSSH2_H_42578934275823624556 +#define INIT_LIBSSH2_H_42578934275823624556 + +namespace zen +{ +void libssh2Init(); //WITHOUT OpenSSL initialization! +void libssh2TearDown(); +} + +#endif //INIT_LIBSSH2_H_42578934275823624556 diff --git a/FreeFileSync/Source/fs/libssh2/init_open_ssl.cpp b/FreeFileSync/Source/fs/libssh2/init_open_ssl.cpp new file mode 100644 index 00000000..a40de204 --- /dev/null +++ b/FreeFileSync/Source/fs/libssh2/init_open_ssl.cpp @@ -0,0 +1,45 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "init_open_ssl.h" +#include <cassert> +#include <openssl/ssl.h> + + +#ifndef OPENSSL_THREADS + #error FFS, we are royally screwed! +#endif + +#if OPENSSL_VERSION_NUMBER < 0x10100000L + #error OpenSSL version too old +#endif + + +void zen::openSslInit() +{ + //official Wiki: https://wiki.openssl.org/index.php/Library_Initialization + //see apps_shutdown(): https://github.com/openssl/openssl/blob/master/apps/openssl.c + //see Curl_ossl_cleanup(): https://github.com/curl/curl/blob/master/lib/vtls/openssl.c + + //excplicitly init OpenSSL on main thread: they seem to initialize atomically! But it still might help to avoid issues: + if (::OPENSSL_init_ssl(OPENSSL_INIT_SSL_DEFAULT, nullptr) != 1) //https://www.openssl.org/docs/man1.1.0/ssl/OPENSSL_init_ssl.html + assert(false); +} + + +void zen::openSslTearDown() {} //OpenSSL 1.1.0+ deprecates all clean up functions + + +struct OpenSslThreadCleanUp +{ + ~OpenSslThreadCleanUp() + { + //OpenSSL 1.1.0+ deprecates all clean up functions + //=> so much the theory, in practice it leaks, of course: https://github.com/openssl/openssl/issues/6283 + OPENSSL_thread_stop(); + } +}; +thread_local OpenSslThreadCleanUp tearDownOpenSslThreadData; diff --git a/FreeFileSync/Source/fs/libssh2/init_open_ssl.h b/FreeFileSync/Source/fs/libssh2/init_open_ssl.h new file mode 100644 index 00000000..e3889a87 --- /dev/null +++ b/FreeFileSync/Source/fs/libssh2/init_open_ssl.h @@ -0,0 +1,16 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef INIT_OPEN_SSL_H_2845370834275264456 +#define INIT_OPEN_SSL_H_2845370834275264456 + +namespace zen +{ +void openSslInit(); +void openSslTearDown(); +} + +#endif //INIT_OPEN_SSL_H_2845370834275264456 diff --git a/FreeFileSync/Source/fs/native.cpp b/FreeFileSync/Source/fs/native.cpp index 9aeddc15..47dd3ff3 100755..100644 --- a/FreeFileSync/Source/fs/native.cpp +++ b/FreeFileSync/Source/fs/native.cpp @@ -145,14 +145,10 @@ ItemDetailsRaw getItemDetails(const Zstring& itemPath) //throw FileError if (::lstat(itemPath.c_str(), &statData) != 0) //lstat() does not resolve symlinks THROW_LAST_FILE_ERROR(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(itemPath)), L"lstat"); - if (S_ISLNK(statData.st_mode)) //on Linux there is no distinction between file and directory symlinks! - return { ItemType::SYMLINK, statData.st_mtime, 0, generateFileId(statData) }; - - else if (S_ISDIR(statData.st_mode)) //a directory - return { ItemType::FOLDER, statData.st_mtime, 0, generateFileId(statData) }; - - else //a file or named pipe, etc. => dont't check using S_ISREG(): see comment in file_traverser.cpp - return { ItemType::FILE, statData.st_mtime, makeUnsigned(statData.st_size), generateFileId(statData) }; + return { S_ISLNK(statData.st_mode) ? ItemType::SYMLINK : //on Linux there is no distinction between file and directory symlinks! + (S_ISDIR(statData.st_mode) ? ItemType::FOLDER : + ItemType::FILE), //a file or named pipe, etc. => dont't check using S_ISREG(): see comment in file_traverser.cpp + statData.st_mtime, makeUnsigned(statData.st_size), generateFileId(statData) }; } ItemDetailsRaw getSymlinkTargetDetails(const Zstring& linkPath) //throw FileError @@ -161,147 +157,101 @@ ItemDetailsRaw getSymlinkTargetDetails(const Zstring& linkPath) //throw FileErro if (::stat(linkPath.c_str(), &statData) != 0) THROW_LAST_FILE_ERROR(replaceCpy(_("Cannot resolve symbolic link %x."), L"%x", fmtPath(linkPath)), L"stat"); - if (S_ISDIR(statData.st_mode)) //a directory - return { ItemType::FOLDER, statData.st_mtime, 0, generateFileId(statData) }; - else //a file or named pipe, etc. - return { ItemType::FILE, statData.st_mtime, makeUnsigned(statData.st_size), generateFileId(statData) }; + return { S_ISDIR(statData.st_mode) ? ItemType::FOLDER : ItemType::FILE, statData.st_mtime, makeUnsigned(statData.st_size), generateFileId(statData) }; } -struct GetDirDetails +class SingleFolderTraverser { - GetDirDetails(const Zstring& dirPath) : dirPath_(dirPath) {} - - using Result = std::vector<FsItemRaw>; - Result operator()() const - { - return getDirContentFlat(dirPath_); //throw FileError - } - -private: - Zstring dirPath_; -}; - - -struct GetItemDetails //details not already retrieved by raw folder traversal -{ - GetItemDetails(const FsItemRaw& rawItem) : rawItem_(rawItem) {} - - struct Result - { - FsItemRaw raw; - ItemDetailsRaw details; - }; - Result operator()() const +public: + SingleFolderTraverser(const std::vector<std::pair<Zstring, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/) { - return { rawItem_, getItemDetails(rawItem_.itemPath) }; //throw FileError - } - -private: - FsItemRaw rawItem_; -}; - + for (const auto& [folderPath, cb] : workload) + workload_.push_back({ folderPath, cb }); -struct GetLinkTargetDetails -{ - GetLinkTargetDetails(const FsItemRaw& rawItem, const ItemDetailsRaw& linkDetails) : rawItem_(rawItem), linkDetails_(linkDetails) {} + while (!workload_.empty()) + { + WorkItem wi = std::move(workload_. back()); //yes, no strong exception guarantee (std::bad_alloc) + /**/ workload_.pop_back(); // - struct Result - { - FsItemRaw raw; - ItemDetailsRaw link; - ItemDetailsRaw target; - }; - Result operator()() const - { - return { rawItem_, linkDetails_, getSymlinkTargetDetails(rawItem_.itemPath) }; //throw FileError + tryReportingDirError([&] //throw X + { + traverseWithException(wi.dirPath, *wi.cb); //throw FileError, X + }, *wi.cb); + } } private: - FsItemRaw rawItem_; - ItemDetailsRaw linkDetails_; -}; - - -void traverseFolderRecursiveNative(const std::vector<std::pair<Zstring, std::shared_ptr<AFS::TraverserCallback>>>& initialTasks /*throw X*/, size_t parallelOps) -{ - std::vector<Task<TravContext, GetDirDetails>> genItems; - - for (const auto& [folderPath, cb] : initialTasks) - genItems.push_back({ GetDirDetails(folderPath), - TravContext{ Zstring() /*errorItemName*/, 0 /*errorRetryCount*/, cb /*TraverserCallback*/ }}); + SingleFolderTraverser (const SingleFolderTraverser&) = delete; + SingleFolderTraverser& operator=(const SingleFolderTraverser&) = delete; - GenericDirTraverser<GetDirDetails, GetItemDetails, GetLinkTargetDetails>(std::move(genItems), parallelOps, "Native Traverser"); //throw X -} -} - - -template <> -template <> -void GenericDirTraverser<GetDirDetails, GetItemDetails, GetLinkTargetDetails>::evalResultValue<GetDirDetails>(const GetDirDetails::Result& r, std::shared_ptr<AFS::TraverserCallback>& cb /*throw X*/) -{ - //attention: if we simply appended to the work queue this would repeatedly allow for situations where a large number of directories are traversed one after another - // without intermittent calls to evalResultValue<GetItemDetails>() => user incorrectly thinks the app is hanging! https://freefilesync.org/forum/viewtopic.php?t=5729 - //solution: *prepend* GetItemDetails() tasks (in correct order) to the work queue ASAP: - std::for_each(r.rbegin(), r.rend(), [&](const FsItemRaw& rawItem) + void traverseWithException(const Zstring& dirPath, AFS::TraverserCallback& cb) //throw FileError, X { - scheduler_.run<GetItemDetails>({ GetItemDetails(rawItem), TravContext{ rawItem.itemName, 0 /*errorRetryCount*/, cb }}, - true /*insertFront*/); - }); -} - - -template <> -template <> -void GenericDirTraverser<GetDirDetails, GetItemDetails, GetLinkTargetDetails>::evalResultValue<GetItemDetails>(const GetItemDetails::Result& r, std::shared_ptr<AFS::TraverserCallback>& cb /*throw X*/) -{ - switch (r.details.type) - { - case ItemType::FILE: - cb->onFile({ r.raw.itemName, r.details.fileSize, r.details.modTime, convertToAbstractFileId(r.details.fileId), nullptr /*symlinkInfo*/ }); //throw X - break; - - case ItemType::FOLDER: - if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb->onFolder({ r.raw.itemName, nullptr /*symlinkInfo*/ })) //throw X - scheduler_.run<GetDirDetails>({ GetDirDetails(r.raw.itemPath), TravContext{ Zstring() /*errorItemName*/, 0 /*errorRetryCount*/, std::move(cbSub) }}); - break; + for (const auto& [itemName, itemPath] : getDirContentFlat(dirPath)) //throw FileError + { + ItemDetailsRaw detailsRaw = {}; + if (!tryReportingItemError([&] //throw X + { + detailsRaw = getItemDetails(itemPath); //throw FileError + }, cb, itemName)) + continue; //ignore error: skip file - case ItemType::SYMLINK: - switch (cb->onSymlink({ r.raw.itemName, r.details.modTime })) //throw X + switch (detailsRaw.type) { - case AFS::TraverserCallback::LINK_FOLLOW: - scheduler_.run<GetLinkTargetDetails>({ GetLinkTargetDetails(r.raw, r.details), TravContext{ r.raw.itemName, 0 /*errorRetryCount*/, cb }}); + case ItemType::FILE: + cb.onFile({ itemName, detailsRaw.fileSize, detailsRaw.modTime, convertToAbstractFileId(detailsRaw.fileId), nullptr /*symlinkInfo*/ }); //throw X break; - case AFS::TraverserCallback::LINK_SKIP: + case ItemType::FOLDER: + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ itemName, nullptr /*symlinkInfo*/ })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + break; + + case ItemType::SYMLINK: + switch (cb.onSymlink({ itemName, detailsRaw.modTime })) //throw X + { + case AFS::TraverserCallback::LINK_FOLLOW: + { + ItemDetailsRaw linkDetails = {}; + if (!tryReportingItemError([&] //throw X + { + linkDetails = getSymlinkTargetDetails(itemPath); //throw FileError + }, cb, itemName)) + continue; + + const AFS::SymlinkInfo linkInfo = { itemName, linkDetails.modTime }; + + if (linkDetails.type == ItemType::FOLDER) + { + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ itemName, &linkInfo })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + } + else //a file or named pipe, etc. + cb.onFile({ itemName, linkDetails.fileSize, linkDetails.modTime, convertToAbstractFileId(linkDetails.fileId), &linkInfo }); //throw X + } + break; + + case AFS::TraverserCallback::LINK_SKIP: + break; + } break; } - break; + } } -} - -template <> -template <> -void GenericDirTraverser<GetDirDetails, GetItemDetails, GetLinkTargetDetails>::evalResultValue<GetLinkTargetDetails>(const GetLinkTargetDetails::Result& r, std::shared_ptr<AFS::TraverserCallback>& cb /*throw X*/) -{ - assert(r.link.type == ItemType::SYMLINK && r.target.type != ItemType::SYMLINK); - - const AFS::SymlinkInfo linkInfo = { r.raw.itemName, r.link.modTime }; - - if (r.target.type == ItemType::FOLDER) + struct WorkItem { - if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb->onFolder({ r.raw.itemName, &linkInfo })) //throw X - scheduler_.run<GetDirDetails>({ GetDirDetails(r.raw.itemPath), TravContext{ Zstring() /*errorItemName*/, 0 /*errorRetryCount*/, std::move(cbSub) }}); - } - else //a file or named pipe, etc. - cb->onFile({ r.raw.itemName, r.target.fileSize, r.target.modTime, convertToAbstractFileId(r.target.fileId), &linkInfo }); //throw X -} + Zstring dirPath; + std::shared_ptr<AFS::TraverserCallback> cb; + }; + std::vector<WorkItem> workload_; +}; -namespace +void traverseFolderRecursiveNative(const std::vector<std::pair<Zstring, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/, size_t) //throw X { - + SingleFolderTraverser dummy(workload); //throw X +} //==================================================================================================== //==================================================================================================== @@ -443,11 +393,13 @@ private: std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const override //throw FileError { - return itemStillExistsViaFolderTraversal(afsPath); //throw FileError + //default implementation: folder traversal + return AbstractFileSystem::itemStillExists(afsPath); //throw FileError } //---------------------------------------------------------------------------------------------------------------- - //target existing: fail/ignore => Native will fail and give a clear error message + //already existing: fail/ignore + //=> Native will fail and give a clear error message void createFolderPlain(const AfsPath& afsPath) const override //throw FileError { initComForThread(); //throw FileError @@ -472,6 +424,14 @@ private: zen::removeDirectoryPlain(getNativePath(afsPath)); //throw FileError } + void removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const override //one call for each object! + { + //default implementation: folder traversal + AbstractFileSystem::removeFolderIfExistsRecursion(afsPath, onBeforeFileDeletion, onBeforeFolderDeletion); //throw FileError, X + } + //---------------------------------------------------------------------------------------------------------------- AbstractPath getSymlinkResolvedPath(const AfsPath& afsPath) const override //throw FileError { @@ -621,10 +581,8 @@ private: catch (FileError&) { assert(false); return ImageHolder(); } } - void connectNetworkFolder(const AfsPath& afsPath, bool allowUserInteraction) const override //throw FileError + void authenticateAccess(bool allowUserInteraction) const override //throw FileError { - //TODO: clean-up/remove/re-think connectNetworkFolder() - } int getAccessTimeout() const override { return 0; } //returns "0" if no timeout in force @@ -693,8 +651,8 @@ bool fff::acceptsItemPathPhraseNative(const Zstring& itemPathPhrase) //noexcept if (startsWith(path, Zstr("["))) //drive letter by volume name syntax return true; - //don't accept relative paths!!! indistinguishable from Explorer MTP paths! - //don't accept paths missing the shared folder! (see drag & drop validation!) + //don't accept relative paths!!! indistinguishable from MTP paths as shown in Explorer's address bar! + //don't accept empty paths (see drag & drop validation!) return static_cast<bool>(parsePathComponents(path)); } @@ -711,6 +669,6 @@ AbstractPath fff::createItemPathNativeNoFormatting(const Zstring& nativePath) // { if (const std::optional<PathComponents> comp = parsePathComponents(nativePath)) return AbstractPath(makeSharedRef<NativeFileSystem>(comp->rootPath), AfsPath(comp->relPath)); - else //path syntax broken + else //broken path syntax return AbstractPath(makeSharedRef<NativeFileSystem>(nativePath), AfsPath()); } diff --git a/FreeFileSync/Source/fs/native.h b/FreeFileSync/Source/fs/native.h index 7c5b004e..7c5b004e 100755..100644 --- a/FreeFileSync/Source/fs/native.h +++ b/FreeFileSync/Source/fs/native.h diff --git a/FreeFileSync/Source/fs/sftp.cpp b/FreeFileSync/Source/fs/sftp.cpp new file mode 100644 index 00000000..ec12b1dd --- /dev/null +++ b/FreeFileSync/Source/fs/sftp.cpp @@ -0,0 +1,2079 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "sftp.h" +#include <zen/sys_error.h> +#include <zen/thread.h> +#include <zen/globals.h> +#include <zen/file_io.h> +#include <zen/basic_math.h> +#include <zen/socket.h> +#include <libssh2_sftp.h> +#include "init_curl_libssh2.h" +#include "ftp_common.h" +#include "abstract_impl.h" +#include "../base/resolve_path.h" + +using namespace zen; +using namespace fff; +using AFS = AbstractFileSystem; + + +namespace +{ +Zstring concatenateSftpFolderPathPhrase(const SftpLoginInfo& login, const AfsPath& afsPath); //noexcept + +/* +SFTP specification version 3 (implemented by libssh2): https://filezilla-project.org/specs/draft-ietf-secsh-filexfer-02.txt + +libssh2: prefer OpenSSL over WinCNG backend: + +WinCNG supports the following ciphers: + rijndael-cbc@lysator.liu.se + aes256-cbc + aes192-cbc + aes128-cbc + arcfour128 + arcfour + 3des-cbc + +OpenSSL supports the same ciphers like WinCNG plus the following: + aes256-ctr + aes192-ctr + aes128-ctr + cast128-cbc + blowfish-cbc +*/ + +const Zchar sftpPrefix[] = Zstr("sftp:"); + +const std::chrono::seconds SFTP_SESSION_MAX_IDLE_TIME (20); +const std::chrono::seconds SFTP_SESSION_CLEANUP_INTERVAL (4); //facilitate default of 5-seconds delay for error retry +const std::chrono::seconds SFTP_CHANNEL_LIMIT_DETECTION_TIME_OUT(30); + +//attention: if operation fails due to time out, e.g. file copy, the cleanup code may hang, too => total delay = 2 x time out interval + +const size_t SFTP_OPTIMAL_BLOCK_SIZE_READ = 4 * MAX_SFTP_READ_SIZE; //https://github.com/libssh2/libssh2/issues/90 +const size_t SFTP_OPTIMAL_BLOCK_SIZE_WRITE = 4 * MAX_SFTP_OUTGOING_SIZE; // +static_assert(MAX_SFTP_READ_SIZE == 30000 && MAX_SFTP_OUTGOING_SIZE == 30000, "reevaluate optimal block sizes if these constants change!"); +/* +Perf Test, Sourceforge frs, SFTP upload, compressed 25 MB test file: + +SFTP_OPTIMAL_BLOCK_SIZE_READ: + multiples of + MAX_SFTP_READ_SIZE kb/s + 1 650 + 2 1000 + 4 1800 + 8 1800 + 16 1800 + 32 1800 + Filezilla download speed: 1800 kb/s + DSL maximum download speed: 3060 kb/s + +SFTP_OPTIMAL_BLOCK_SIZE_WRITE: + multiples of + MAX_SFTP_OUTGOING_SIZE kb/s + 1 140 + 2 280 + 4 320 + 8 320 + 16 320 + 32 320 + Filezilla upload speed: 560 kb/s + DSL maximum upload speed: 620 kb/s + +=> first call to libssh2_sftp_read/libssh2_sftp_write may take quite long for 16x and larger => use smallest multiple that fills bandwidth! +*/ + + +//use all configuration data that *defines* an SSH session as key when buffering sessions! This is what user expects, e.g. when changing settings in SFTP login dialog +struct SshSessionId +{ + /*explicit*/ SshSessionId(const SftpLoginInfo& login) : + server(login.server), + port(login.port), + username(login.username), + authType(login.authType), + password(login.password), + privateKeyFilePath(login.privateKeyFilePath) {} + + Zstring server; + int port = 0; + Zstring username; + SftpAuthType authType = SftpAuthType::PASSWORD; + Zstring password; + Zstring privateKeyFilePath; + //traverserChannelsPerConnection => irrelevant for session equality + //timeoutSec +}; + + +bool operator<(const SshSessionId& lhs, const SshSessionId& rhs) +{ + //exactly the type of case insensitive comparison we need for server names! + int rv = compareAsciiNoCase(lhs.server, rhs.server); //https://msdn.microsoft.com/en-us/library/windows/desktop/ms738519#IDNs + if (rv != 0) + return rv < 0; + + if (lhs.port != rhs.port) + return lhs.port < rhs.port; + + rv = compareString(lhs.username, rhs.username); //case sensitive! + if (rv != 0) + return rv < 0; + + if (lhs.authType != rhs.authType) + return lhs.authType < rhs.authType; + + switch (lhs.authType) + { + case SftpAuthType::PASSWORD: + return compareString(lhs.password, rhs.password) < 0; //case sensitive! + + case SftpAuthType::KEY_FILE: + rv = compareString(lhs.password, rhs.password); //case sensitive! + if (rv != 0) + return rv < 0; + + return compareString(lhs.privateKeyFilePath, rhs.privateKeyFilePath) < 0; //case sensitive! + + case SftpAuthType::AGENT: + return false; + } + assert(false); + return false; +} + + +std::string getLibssh2Path(const AfsPath& afsPath) +{ + return utfTo<std::string>(getServerRelPath(afsPath)); +} + + +std::wstring getSftpDisplayPath(const Zstring& serverName, const AfsPath& afsPath) +{ + Zstring displayPath = Zstring(sftpPrefix) + Zstr("//") + serverName; + const Zstring relPath = getServerRelPath(afsPath); + if (relPath != Zstr("/")) + displayPath += relPath; + return utfTo<std::wstring>(displayPath); +} +//don't show username and password! + +//=========================================================================================================================== + +std::wstring formatSshErrorRaw(int ec) +{ + switch (ec) + { + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_NONE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SOCKET_NONE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_BANNER_RECV); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_BANNER_SEND); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_INVALID_MAC); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_KEX_FAILURE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_ALLOC); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SOCKET_SEND); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_KEY_EXCHANGE_FAILURE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_TIMEOUT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_HOSTKEY_INIT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_HOSTKEY_SIGN); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_DECRYPT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SOCKET_DISCONNECT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_PROTO); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_PASSWORD_EXPIRED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_FILE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_METHOD_NONE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_AUTHENTICATION_FAILED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_PUBLICKEY_UNVERIFIED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_OUTOFORDER); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_FAILURE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_REQUEST_DENIED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_UNKNOWN); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_WINDOW_EXCEEDED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_PACKET_EXCEEDED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_CLOSED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_CHANNEL_EOF_SENT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SCP_PROTOCOL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_ZLIB); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SOCKET_TIMEOUT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SFTP_PROTOCOL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_REQUEST_DENIED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_METHOD_NOT_SUPPORTED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_INVAL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_INVALID_POLL_TYPE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_PUBLICKEY_PROTOCOL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_EAGAIN); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_BUFFER_TOO_SMALL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_BAD_USE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_COMPRESS); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_OUT_OF_BOUNDARY); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_AGENT_PROTOCOL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_SOCKET_RECV); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_ENCRYPT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_BAD_SOCKET); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_ERROR_KNOWN_HOSTS); + } + return L"Unknown SSH error: " + numberTo<std::wstring>(ec); +} + +std::wstring formatSftpErrorRaw(unsigned long ec) +{ + switch (ec) + { + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_OK); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_EOF); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NO_SUCH_FILE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_PERMISSION_DENIED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_FAILURE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_BAD_MESSAGE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NO_CONNECTION); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_CONNECTION_LOST); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_OP_UNSUPPORTED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_INVALID_HANDLE); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NO_SUCH_PATH); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_FILE_ALREADY_EXISTS); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_WRITE_PROTECT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NO_MEDIA); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NO_SPACE_ON_FILESYSTEM); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_QUOTA_EXCEEDED); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_UNKNOWN_PRINCIPAL); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_LOCK_CONFLICT); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_DIR_NOT_EMPTY); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_NOT_A_DIRECTORY); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_INVALID_FILENAME); + ZEN_CHECK_CASE_FOR_CONSTANT(LIBSSH2_FX_LINK_LOOP); + + //SFTP error codes missing from libssh2: http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13#section-9.1 + case 22: + return L"SSH_FX_CANNOT_DELETE"; + case 23: + return L"SSH_FX_INVALID_PARAMETER"; + case 24: + return L"SSH_FX_FILE_IS_A_DIRECTORY"; + case 25: + return L"SSH_FX_BYTE_RANGE_LOCK_CONFLICT"; + case 26: + return L"SSH_FX_BYTE_RANGE_LOCK_REFUSED"; + case 27: + return L"SSH_FX_DELETE_PENDING"; + case 28: + return L"SSH_FX_FILE_CORRUPT"; + case 29: + return L"SSH_FX_OWNER_INVALID"; + case 30: + return L"SSH_FX_GROUP_INVALID"; + case 31: + return L"SSH_FX_NO_MATCHING_BYTE_RANGE_LOCK"; + } + return L"Unknown SFTP error: " + numberTo<std::wstring>(ec); +} + + +std::wstring formatLastSshError(const std::wstring& functionName, LIBSSH2_SESSION* sshSession, LIBSSH2_SFTP* sftpChannel /*optional*/) +{ + char* lastErrorMsg = nullptr; //owned by "sshSession" + const int lastErrorCode = ::libssh2_session_last_error(sshSession, &lastErrorMsg, nullptr, false /*want_buf*/); + assert(lastErrorMsg); + + std::wstring errorMsg; + if (lastErrorMsg) + errorMsg = trimCpy(utfTo<std::wstring>(lastErrorMsg)); + + if (sftpChannel && lastErrorCode == LIBSSH2_ERROR_SFTP_PROTOCOL) + errorMsg += (errorMsg.empty() ? L"" : L" - ") + formatSftpErrorRaw(::libssh2_sftp_last_error(sftpChannel)); + + return formatSystemError(functionName, formatSshErrorRaw(lastErrorCode), errorMsg); +} + +//=========================================================================================================================== + +class FatalSshError //=> consider SshSession corrupted and stop use ASAP! same conceptual level like FileError +{ +public: + FatalSshError(const std::wstring& msg, const std::wstring& details) : msg_(msg + L"\n\n" + details) {} + const std::wstring& toString() const { return msg_; } + +private: + std::wstring msg_; +}; + + +Global<UniSessionCounter> globalSftpSessionCount(createUniSessionCounter()); + + +class SshSession //throw FileError +{ +public: + SshSession(const SshSessionId& sessionId, int timeoutSec) : sessionId_(sessionId) //throw FileError + { + ZEN_ON_SCOPE_FAIL(cleanup()); //destructor call would lead to member double clean-up!!! + + try + { + libsshCurlUnifiedInitCookie_ = getLibsshCurlUnifiedInitCookie(globalSftpSessionCount); //throw SysError + + Zstring serviceName = Zstr("ssh"); //SFTP default port: 22, see %WINDIR%\system32\drivers\etc\services + if (sessionId_.port > 0) + serviceName = numberTo<Zstring>(sessionId_.port); + + socket_ = std::make_unique<Socket>(sessionId_.server, serviceName); //throw SysError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + + sshSession_ = ::libssh2_session_init(); + if (!sshSession_) //does not set ssh last error; source: only memory allocation may fail + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(L"libssh2_session_init", formatSshErrorRaw(LIBSSH2_ERROR_ALLOC), std::wstring())); + + /* + => libssh2 using zlib crashes for Bitvise Servers: https://freefilesync.org/forum/viewtopic.php?t=2825 + => Don't enable zlib compression: libssh2 also recommends this option disabled: http://comments.gmane.org/gmane.network.ssh.libssh2.devel/6203 + const int rc = ::libssh2_session_flag(sshSession_, LIBSSH2_FLAG_COMPRESS, 1); //does not set ssh last error + if (rc != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatSystemError(L"libssh2_session_flag", formatSshErrorRaw(rc), std::wstring())); + => build libssh2 without LIBSSH2_HAVE_ZLIB + */ + + ::libssh2_session_set_blocking(sshSession_, 1); + + //we don't consider the timeout part of the session when it comes to reuse! but we already require it during initialization + ::libssh2_session_set_timeout(sshSession_, timeoutSec * 1000 /*ms*/); + + + + + if (::libssh2_session_handshake(sshSession_, socket_->get()) != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_session_handshake", sshSession_, nullptr)); + + //evaluate fingerprint = libssh2_hostkey_hash(sshSession_, LIBSSH2_HOSTKEY_HASH_SHA1) ??? + + const auto usernameUtf8 = utfTo<std::string>(sessionId_.username); + const auto passwordUtf8 = utfTo<std::string>(sessionId_.password); + + const char* authList = ::libssh2_userauth_list(sshSession_, usernameUtf8.c_str(), static_cast<unsigned int>(usernameUtf8.length())); + if (!authList) + { + if (::libssh2_userauth_authenticated(sshSession_) == 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_userauth_list", sshSession_, nullptr)); + //else: SSH_USERAUTH_NONE has authenticated successfully => we're already done + } + else + { + bool supportAuthPassword = false; + bool supportAuthKeyfile = false; + bool supportAuthInteractive = false; + for (const std::string& str : split<std::string>(authList, ',', SplitType::SKIP_EMPTY)) + { + const std::string authMethod = trimCpy(str); + if (authMethod == "password") + supportAuthPassword = true; + else if (authMethod == "publickey") + supportAuthKeyfile = true; + else if (authMethod == "keyboard-interactive") + supportAuthInteractive = true; + } + + switch (sessionId_.authType) + { + case SftpAuthType::PASSWORD: + { + if (supportAuthPassword) + { + if (::libssh2_userauth_password(sshSession_, usernameUtf8.c_str(), passwordUtf8.c_str()) != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_userauth_password", sshSession_, nullptr)); + } + else if (supportAuthInteractive) //some servers, e.g. web.sourceforge.net, support "keyboard-interactive", but not "password" + { + std::wstring unexpectedPrompts; + + auto authCallback = [&](int num_prompts, const LIBSSH2_USERAUTH_KBDINT_PROMPT* prompts, LIBSSH2_USERAUTH_KBDINT_RESPONSE* responses) + { + //note: FileZilla assumes password requests when it finds "num_prompts == 1" and "!echo" -> prompt may be localized! + //test case: sourceforge.net sends a single "Password: " prompt with "!echo" + if (num_prompts == 1 && prompts[0].echo == 0) + { + responses[0].text = //pass ownership; will be ::free()d + ::strdup(passwordUtf8.c_str()); + responses[0].length = static_cast<unsigned int>(passwordUtf8.size()); + } + else + for (int i = 0; i < num_prompts; ++i) + unexpectedPrompts += (unexpectedPrompts.empty() ? L"" : L"|") + utfTo<std::wstring>(std::string(prompts[i].text, prompts[i].length)); + }; + using AuthCbType = decltype(authCallback); + + auto authCallbackWrapper = [](const char* name, int name_len, const char* instruction, int instruction_len, + int num_prompts, const LIBSSH2_USERAUTH_KBDINT_PROMPT* prompts, LIBSSH2_USERAUTH_KBDINT_RESPONSE* responses, void** abstract) + { + try + { + AuthCbType* callback = *reinterpret_cast<AuthCbType**>(abstract); //free this poor little C-API from its shackles and redirect to a proper lambda + (*callback)(num_prompts, prompts, responses); //name, instruction are nullptr for sourceforge.net + } + catch (...) { assert(false); } + }; + + if (*::libssh2_session_abstract(sshSession_)) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), L"libssh2_session_abstract: non-null value"); + + *reinterpret_cast<AuthCbType**>(::libssh2_session_abstract(sshSession_)) = &authCallback; + ZEN_ON_SCOPE_EXIT(*::libssh2_session_abstract(sshSession_) = nullptr); + + if (::libssh2_userauth_keyboard_interactive(sshSession_, usernameUtf8.c_str(), authCallbackWrapper) != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + formatLastSshError(L"libssh2_userauth_keyboard_interactive", sshSession_, nullptr) + + (unexpectedPrompts.empty() ? L"" : L"\nUnexpected prompts: " + unexpectedPrompts)); + } + else + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + replaceCpy(_("The server does not support authentication via %x."), L"%x", L"\"username/password\"") + + L"\n" +_("Required:") + L" " + utfTo<std::wstring>(authList)); + } + break; + + case SftpAuthType::KEY_FILE: + { + if (!supportAuthKeyfile) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), + replaceCpy(_("The server does not support authentication via %x."), L"%x", L"\"key file\"") + + L"\n" +_("Required:") + L" " + utfTo<std::wstring>(authList)); + + std::string pkStream; + try + { + pkStream = loadBinContainer<std::string>(sessionId_.privateKeyFilePath, nullptr /*notifyUnbufferedIO*/); //throw FileError + } + catch (const FileError& e) { throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), e.toString()); } + + if (::libssh2_userauth_publickey_frommemory(sshSession_, //LIBSSH2_SESSION *session, + usernameUtf8.c_str(), //const char *username, + usernameUtf8.size(), //size_t username_len, + nullptr, //const char *publickeydata, + 0, //size_t publickeydata_len, + pkStream.c_str(), //const char *privatekeydata, + pkStream.size(), //size_t privatekeydata_len, + passwordUtf8.c_str()) != 0) //const char *passphrase + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_userauth_publickey_frommemory", sshSession_, nullptr)); + } + break; + + case SftpAuthType::AGENT: + { + LIBSSH2_AGENT* sshAgent = ::libssh2_agent_init(sshSession_); + if (!sshAgent) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_agent_init", sshSession_, nullptr)); + ZEN_ON_SCOPE_EXIT(::libssh2_agent_free(sshAgent)); + + if (::libssh2_agent_connect(sshAgent) != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_agent_connect", sshSession_, nullptr)); + ZEN_ON_SCOPE_EXIT(::libssh2_agent_disconnect(sshAgent)); + + if (::libssh2_agent_list_identities(sshAgent) != 0) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_agent_list_identities", sshSession_, nullptr)); + + for (libssh2_agent_publickey* prev = nullptr;;) + { + libssh2_agent_publickey* identity = nullptr; + const int rc = ::libssh2_agent_get_identity(sshAgent, &identity, prev); + if (rc == 0) //public key returned + ; + else if (rc == 1) //no more public keys + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), L"SSH agent contains no matching public key."); + else + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sessionId_.server)), formatLastSshError(L"libssh2_agent_get_identity", sshSession_, nullptr)); + + if (::libssh2_agent_userauth(sshAgent, usernameUtf8.c_str(), identity) == 0) + break; //authentication successful + + //else: failed => try next public key + prev = identity; + } + } + break; + } + } + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + } + + ~SshSession() { cleanup(); } + + const SshSessionId& getSessionId() const { return sessionId_; } + + bool isHealthy() const + { + for (const SftpChannelInfo& ci : sftpChannels_) + if (ci.nbInfo.commandPending) + return false; + + if (nbInfo_.commandPending) + return false; + + if (possiblyCorrupted_) + return false; + + if (numeric::dist(std::chrono::steady_clock::now(), lastSuccessfulUseTime_) > SFTP_SESSION_MAX_IDLE_TIME) + return false; + + return true; + } + + void markAsCorrupted() { possiblyCorrupted_ = true; } + + struct Details + { + LIBSSH2_SESSION* sshSession; + LIBSSH2_SFTP* sftpChannel; + }; + + size_t getSftpChannelCount() const { return sftpChannels_.size(); } + + //return "false" if pending + bool tryNonBlocking(size_t channelNo, std::chrono::steady_clock::time_point commandStartTime, const std::wstring& functionName, + const std::function<int(const SshSession::Details& sd)>& sftpCommand /*noexcept!*/, int timeoutSec) //throw SysError, FatalSshError + { + assert(::libssh2_session_get_blocking(sshSession_)); + ::libssh2_session_set_blocking(sshSession_, 0); + ZEN_ON_SCOPE_EXIT(::libssh2_session_set_blocking(sshSession_, 1)); + + //yes, we're non-blocking, still won't hurt to set the timeout in case libssh2 decides to use it nevertheless + ::libssh2_session_set_timeout(sshSession_, timeoutSec * 1000 /*ms*/); + + LIBSSH2_SFTP* sftpChannel = channelNo < sftpChannels_.size() ? sftpChannels_[channelNo].sftpChannel : nullptr; + SftpNonBlockInfo& nbInfo = channelNo < sftpChannels_.size() ? sftpChannels_[channelNo].nbInfo : nbInfo_; + + if (!nbInfo.commandPending) + assert(nbInfo.commandStartTime != commandStartTime); + else if (nbInfo.commandStartTime == commandStartTime && nbInfo.functionName == functionName) + ; //continue pending SFTP call + else + { + assert(false); //pending sftp command is not completed by client: e.g. libssh2_sftp_close() cleaning up after a timed-out libssh2_sftp_read() + possiblyCorrupted_ = true; //=> start new command (with new start time), but remember to not trust this session anymore! + } + nbInfo.commandPending = true; + nbInfo.commandStartTime = commandStartTime; + nbInfo.functionName = functionName; + + int rc = LIBSSH2_ERROR_NONE; + try + { + rc = sftpCommand({ sshSession_, sftpChannel }); //noexcept + } + catch (...) { assert(false); rc = LIBSSH2_ERROR_BAD_USE; } + + assert(rc >= 0 || ::libssh2_session_last_errno(sshSession_) == rc); + if (rc < 0 && ::libssh2_session_last_errno(sshSession_) != rc) //just in case libssh2 failed to properly set last error; e.g. https://github.com/libssh2/libssh2/pull/123 + ::libssh2_session_set_last_error(sshSession_, rc, nullptr); + + //note: even when non-blocking, libssh2 may return LIBSSH2_ERROR_TIMEOUT, but this seems to be an ordinary error + + if (rc == LIBSSH2_ERROR_EAGAIN) + { + if (numeric::dist(std::chrono::steady_clock::now(), nbInfo.commandStartTime) > std::chrono::seconds(timeoutSec)) + //consider SSH session corrupted! => isHealthy() will see pending command + throw FatalSshError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(sessionId_.server)), + formatSystemError(functionName, formatSshErrorRaw(LIBSSH2_ERROR_TIMEOUT), + _P("Operation timed out after 1 second.", "Operation timed out after %x seconds.", timeoutSec))); + return false; + } + + nbInfo.commandPending = false; + + if (rc < 0) + throw SysError(formatLastSshError(functionName, sshSession_, sftpChannel)); + + lastSuccessfulUseTime_ = std::chrono::steady_clock::now(); + return true; + } + + //returns when traffic is available or time out: both cases are handled by next tryNonBlocking() call + static void waitForTraffic(const std::vector<SshSession*>& sshSessions, int timeoutSec) //throw FatalSshError + { + //reference: session.c: _libssh2_wait_socket() + + if (sshSessions.empty()) return; + + if (sshSessions.size() > FD_SETSIZE) //precise: this limit is for both fd_set containers *each*! + throw FatalSshError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(sshSessions[0]->sessionId_.server)), + _P("Cannot wait on more than 1 connection at a time.", "Cannot wait on more than %x connections at a time.", FD_SETSIZE) + L" " + + replaceCpy(_("Active connections: %x"), L"%x", numberTo<std::wstring>(sshSessions.size()))); + SocketType nfds = 0; + fd_set rfd = {}; + fd_set wfd = {}; + FD_ZERO(&wfd); + FD_ZERO(&rfd); + + fd_set* writefds = nullptr; + fd_set* readfds = nullptr; + + std::chrono::steady_clock::time_point startTimeMax; + + for (SshSession* session : sshSessions) + { + assert(::libssh2_session_last_errno(session->sshSession_) == LIBSSH2_ERROR_EAGAIN); + assert(session->nbInfo_.commandPending || std::any_of(session->sftpChannels_.begin(), session->sftpChannels_.end(), [](SftpChannelInfo& ci) { return ci.nbInfo.commandPending; })); + + const int dir = ::libssh2_session_block_directions(session->sshSession_); + assert(dir != 0); + if (dir & LIBSSH2_SESSION_BLOCK_INBOUND) + { + nfds = std::max(nfds, session->socket_->get()); + FD_SET(session->socket_->get(), &rfd); + readfds = &rfd; + } + if (dir & LIBSSH2_SESSION_BLOCK_OUTBOUND) + { + nfds = std::max(nfds, session->socket_->get()); + FD_SET(session->socket_->get(), &wfd); + writefds = &wfd; + } + + for (SftpChannelInfo& ci : session->sftpChannels_) + if (ci.nbInfo.commandPending) + startTimeMax = std::max(startTimeMax, ci.nbInfo.commandStartTime); + if (session->nbInfo_.commandPending) + startTimeMax = std::max(startTimeMax, session->nbInfo_.commandStartTime); + } + assert(readfds || writefds); + if (!readfds && !writefds) + return; + + assert(startTimeMax != std::chrono::steady_clock::time_point()); + const auto endTime = startTimeMax + std::chrono::seconds(timeoutSec); + const auto now = std::chrono::steady_clock::now(); + + if (now > endTime) + return; //time-out! => let next tryNonBlocking() call fail with detailed error! + const auto waitTimeMs = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - now).count(); + + struct ::timeval tv = {}; + tv.tv_sec = static_cast<long>(waitTimeMs / 1000); + tv.tv_usec = static_cast<long>(waitTimeMs - tv.tv_sec * 1000) * 1000; + + //WSAPoll broken, even ::poll() on OS X? https://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/ + //perf: no significant difference compared to ::WSAPoll() + const int rc = ::select(nfds + 1, readfds, writefds, nullptr /*errorfds*/, &tv); + if (rc == 0) + return; //time-out! => let next tryNonBlocking() call fail with detailed error! + if (rc < 0) + { + //consider SSH sessions corrupted! => isHealthy() will see pending commands + ErrorCode ec = getLastError(); //copy before directly/indirectly making other system calls! + throw FatalSshError(replaceCpy(_("Unable to access %x."), L"%x", fmtPath(sshSessions[0]->sessionId_.server)), formatSystemError(L"select", ec)); + } + } + + static void addSftpChannel(const std::vector<SshSession*>& sshSessions, int timeoutSec) //throw FileError, FatalSshError + { + auto getErrorMsg = [](SshSession& sshSession) //when hitting the server's SFTP channel limit, inform user about channel number + { + std::wstring errorMsg = replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(sshSession.sessionId_.server)); + if (!sshSession.sftpChannels_.empty()) + errorMsg += L" " + replaceCpy(_("Failed to open SFTP channel number %x."), L"%x", numberTo<std::wstring>(sshSession.sftpChannels_.size() + 1)); + return errorMsg; + }; + + std::optional<FileError> firstFileError; + std::optional<FatalSshError> firstFatalError; + + std::vector<SshSession*> pendingSessions = sshSessions; + const auto sftpCommandStartTime = std::chrono::steady_clock::now(); + + for (;;) + { + //create all SFTP sessions in parallel => non-blocking + //note: each libssh2_sftp_init() consists of multiple round-trips => poll until all sessions are finished, don't just init and then block on each! + for (size_t pos = pendingSessions.size(); pos-- > 0 ; ) //CAREFUL WITH THESE ERASEs (invalidate positions!!!) + try + { + if (pendingSessions[pos]->tryNonBlocking(static_cast<size_t>(-1), sftpCommandStartTime, L"libssh2_sftp_init", + [&](const SshSession::Details& sd) //noexcept! + { + LIBSSH2_SFTP* sftpChannelNew = ::libssh2_sftp_init(sd.sshSession); + if (!sftpChannelNew) + return std::min(::libssh2_session_last_errno(sd.sshSession), LIBSSH2_ERROR_SOCKET_NONE); + //just in case libssh2 failed to properly set last error; e.g. https://github.com/libssh2/libssh2/pull/123 + + pendingSessions[pos]->sftpChannels_.emplace_back(sftpChannelNew); + return LIBSSH2_ERROR_NONE; + }, timeoutSec)) //throw SysError, FatalSshError + pendingSessions.erase(pendingSessions.begin() + pos); //= not pending + } + catch (const SysError& e) + { + if (!firstFileError) //don't throw yet and corrupt other valid, but pending SshSessions! We also don't want to leak LIBSSH2_SFTP* waiting in libssh2 code + firstFileError = FileError(getErrorMsg(*pendingSessions[pos]), e.toString()); + pendingSessions.erase(pendingSessions.begin() + pos); + } + catch (const FatalSshError& e) + { + if (!firstFatalError) + firstFatalError = e; + pendingSessions.erase(pendingSessions.begin() + pos); + } + + if (pendingSessions.empty()) + { + if (firstFatalError) //throw FatalSshError *before* FileError (later can be retried) + throw* firstFatalError; + if (firstFileError) + throw* firstFileError; + return; + } + + waitForTraffic(pendingSessions, timeoutSec); //throw FatalSshError + } + } + +private: + SshSession (const SshSession&) = delete; + SshSession& operator=(const SshSession&) = delete; + + void cleanup() + { + //attention: following calls may block heavily on error! Good news: our dedicated cleanup thread will run the ~SshSession + for (SftpChannelInfo& ci : sftpChannels_) + { + assert(!ci.nbInfo.commandPending); //may legitimately happen when an SFTP command times out + ::libssh2_sftp_shutdown(ci.sftpChannel); + } + + if (sshSession_) + { + assert(!nbInfo_.commandPending); + ::libssh2_session_disconnect(sshSession_, "FreeFileSync says \"bye\"!"); + ::libssh2_session_free(sshSession_); + } + } + + struct SftpNonBlockInfo + { + bool commandPending = false; + std::chrono::steady_clock::time_point commandStartTime; //specified by client, try to detect libssh2 usage errors + std::wstring functionName; + }; + + struct SftpChannelInfo + { + explicit SftpChannelInfo(LIBSSH2_SFTP* sc) : sftpChannel(sc) {} + + LIBSSH2_SFTP* sftpChannel = nullptr; + SftpNonBlockInfo nbInfo; + }; + + std::unique_ptr<Socket> socket_; //*bound* after constructor has run + LIBSSH2_SESSION* sshSession_ = nullptr; + std::vector<SftpChannelInfo> sftpChannels_; + bool possiblyCorrupted_ = false; + + SftpNonBlockInfo nbInfo_; //for SSH session, e.g. libssh2_sftp_init() + + const SshSessionId sessionId_; + std::shared_ptr<UniCounterCookie> libsshCurlUnifiedInitCookie_; + std::chrono::steady_clock::time_point lastSuccessfulUseTime_; +}; + +//=========================================================================================================================== +//=========================================================================================================================== + +class SftpSessionManager //reuse (healthy) SFTP sessions globally +{ + struct IdleSshSessions; + +public: + SftpSessionManager() : sessionCleaner_([this] + { + setCurrentThreadName("Session Cleaner[SFTP]"); + runGlobalSessionCleanUp(); /*throw ThreadInterruption*/ + }) {} + + ~SftpSessionManager() + { + sessionCleaner_.interrupt(); + sessionCleaner_.join(); + } + + struct ReUseOnDelete + { + void operator()(SshSession* s) const; + }; + + class SshSessionShared + { + public: + SshSessionShared(std::unique_ptr<SshSession, ReUseOnDelete>&& idleSession, int timeoutSec) : session_(std::move(idleSession)), //bound! + timeoutSec_(timeoutSec) { /*assert(session_->isHealthy());*/ } + + //we need two-step initialization: 1. constructor is FAST and noexcept 2. init() is SLOW and throws + void init() //throw FileError, FatalSshError + { + if (session_->getSftpChannelCount() == 0) //make sure the SSH session contains at least one SFTP channel + SshSession::addSftpChannel({ session_.get() }, timeoutSec_); //throw FileError, FatalSshError + } + + //bool isHealthy() const { return session_->isHealthy(); } + + void executeBlocking(const std::wstring& functionName, const std::function<int(const SshSession::Details& sd)>& sftpCommand /*noexcept!*/) //throw SysError, FatalSshError + { + assert(threadId_ == getThreadId()); + assert(session_->getSftpChannelCount() > 0); + const auto sftpCommandStartTime = std::chrono::steady_clock::now(); + + for (;;) + if (session_->tryNonBlocking(0, sftpCommandStartTime, functionName, sftpCommand, timeoutSec_)) //throw SysError, FatalSshError + return; + else //pending + SshSession::waitForTraffic({ session_.get() }, timeoutSec_); //throw FatalSshError + } + + private: + std::unique_ptr<SshSession, ReUseOnDelete> session_; //bound! + const uint64_t threadId_ = getThreadId(); + const int timeoutSec_; + }; + + class SshSessionExclusive + { + public: + SshSessionExclusive(std::unique_ptr<SshSession, ReUseOnDelete>&& idleSession, int timeoutSec) : session_(std::move(idleSession)), //bound! + timeoutSec_(timeoutSec) { /*assert(session_->isHealthy());*/ } + + bool tryNonBlocking(size_t channelNo, std::chrono::steady_clock::time_point commandStartTime, const std::wstring& functionName, //throw SysError, FatalSshError + const std::function<int(const SshSession::Details& sd)>& sftpCommand /*noexcept!*/) + { + return session_->tryNonBlocking(channelNo, commandStartTime, functionName, sftpCommand, timeoutSec_); //throw SysError, FatalSshError + } + + void finishBlocking(size_t channelNo, std::chrono::steady_clock::time_point commandStartTime, const std::wstring& functionName, + const std::function<int(const SshSession::Details& sd)>& sftpCommand /*noexcept!*/) + { + for (;;) + try + { + if (session_->tryNonBlocking(channelNo, commandStartTime, functionName, sftpCommand, timeoutSec_)) //throw SysError, FatalSshError + return; + else //pending + SshSession::waitForTraffic({ session_.get() }, timeoutSec_); //throw FatalSshError + } + catch (const SysError& ) { return; } + catch (const FatalSshError&) { return; } + } + + size_t getSftpChannelCount() const { return session_->getSftpChannelCount(); } + void markAsCorrupted() { session_->markAsCorrupted(); } + + static void addSftpChannel(const std::vector<SshSessionExclusive*>& exSessions) //throw FileError, FatalSshError + { + std::vector<SshSession*> sshSessions; + for (SshSessionExclusive* exSession : exSessions) + sshSessions.push_back(exSession->session_.get()); + + int timeoutSec = 0; + for (SshSessionExclusive* exSession : exSessions) + timeoutSec = std::max(timeoutSec, exSession->timeoutSec_); + + SshSession::addSftpChannel(sshSessions, timeoutSec); //throw FileError, FatalSshError + } + + static void waitForTraffic(const std::vector<SshSessionExclusive*>& exSessions) //throw FatalSshError + { + std::vector<SshSession*> sshSessions; + for (SshSessionExclusive* exSession : exSessions) + sshSessions.push_back(exSession->session_.get()); + + int timeoutSec = 0; + for (SshSessionExclusive* exSession : exSessions) + timeoutSec = std::max(timeoutSec, exSession->timeoutSec_); + + SshSession::waitForTraffic(sshSessions, timeoutSec); //throw FatalSshError + } + + Zstring getServerName() const { return session_->getSessionId().server; } + + private: + std::unique_ptr<SshSession, ReUseOnDelete> session_; //bound! + const int timeoutSec_; + }; + + + std::shared_ptr<SshSessionShared> getSharedSession(const SftpLoginInfo& login) //throw FileError + { + Protected<IdleSshSessions>& sessionStore = getSessionStore(login); + + const uint64_t threadId = getThreadId(); + std::shared_ptr<SshSessionShared> sharedSession; //no need to protect against concurrency: same thread! + + sessionStore.access([&](IdleSshSessions& sessions) + { + std::weak_ptr<SshSessionShared>& sharedSessionWeak = sessions.sshSessionsWithThreadAffinity[threadId]; //get or create + if (auto session = sharedSessionWeak.lock()) + //dereference session ONLY after affinity to THIS thread was confirmed!!! + //assume "isHealthy()" to avoid hitting server connection limits: (clean up of !isHealthy() after use; idle sessions via worker thread) + sharedSession = session; + + if (!sharedSession) + //assume "isHealthy()" to avoid hitting server connection limits: (clean up of !isHealthy() after use; idle sessions via worker thread) + if (!sessions.idleSshSessions.empty()) + { + std::unique_ptr<SshSession, ReUseOnDelete> sshSession(sessions.idleSshSessions.back().release()); + /**/ sessions.idleSshSessions.pop_back(); + sharedSessionWeak = sharedSession = std::make_shared<SshSessionShared>(std::move(sshSession), login.timeoutSec); //still holding lock => constructor must be *fast*! + } + }); + + //create new SFTP session outside the lock: 1. don't block other threads 2. non-atomic regarding "sessionStore"! => one session too many is not a problem! + if (!sharedSession) + { + sharedSession = std::make_shared<SshSessionShared>(std::unique_ptr<SshSession, ReUseOnDelete>(new SshSession(login, login.timeoutSec)), login.timeoutSec); //throw FileError + sessionStore.access([&](IdleSshSessions& sessions) + { + sessions.sshSessionsWithThreadAffinity[threadId] = sharedSession; + }); + } + + //finish two-step initialization outside the lock: SLOW! + try + { + sharedSession->init(); //throw FileError, FatalSshError + } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //session corrupted => is not returned => no special handling required => FileError is sufficient + + return sharedSession; + } + + + std::unique_ptr<SshSessionExclusive> getExclusiveSession(const SftpLoginInfo& login) //throw FileError + { + Protected<IdleSshSessions>& sessionStore = getSessionStore(login); + + std::unique_ptr<SshSession, ReUseOnDelete> sshSession; + + sessionStore.access([&](IdleSshSessions& sessions) + { + //assume "isHealthy()" to avoid hitting server connection limits: (clean up of !isHealthy() after use, idle sessions via worker thread) + if (!sessions.idleSshSessions.empty()) + { + sshSession.reset(sessions.idleSshSessions.back().release()); + /**/ sessions.idleSshSessions.pop_back(); + } + }); + + //create new SFTP session outside the lock: 1. don't block other threads 2. non-atomic regarding "sessionStore"! => one session too many is not a problem! + if (!sshSession) + sshSession.reset(new SshSession(login, login.timeoutSec)); //throw FileError + + return std::make_unique<SshSessionExclusive>(std::move(sshSession), login.timeoutSec); //throw FileError + } + +private: + SftpSessionManager (const SftpSessionManager&) = delete; + SftpSessionManager& operator=(const SftpSessionManager&) = delete; + + Protected<IdleSshSessions>& getSessionStore(const SshSessionId& sessionId) + { + //single global session store per login; life-time bound to globalInstance => never remove a sessionStore!!! + Protected<IdleSshSessions>* store = nullptr; + + globalSessionStore_.access([&](GlobalSshSessions& sessionsById) + { + store = &sessionsById[sessionId]; //get or create + }); + static_assert(std::is_same_v<GlobalSshSessions, std::map<SshSessionId, Protected<IdleSshSessions>>>, "require std::map so that the pointers we return remain stable"); + + return *store; + } + + //run a dedicated clean-up thread => it's unclear when the server let's a connection time out, so we do it preemptively + //context of worker thread: + void runGlobalSessionCleanUp() //throw ThreadInterruption + { + std::chrono::steady_clock::time_point lastCleanupTime; + for (;;) + { + const auto now = std::chrono::steady_clock::now(); + + if (now < lastCleanupTime + SFTP_SESSION_CLEANUP_INTERVAL) + interruptibleSleep(lastCleanupTime + SFTP_SESSION_CLEANUP_INTERVAL - now); //throw ThreadInterruption + + lastCleanupTime = std::chrono::steady_clock::now(); + + std::vector<Protected<IdleSshSessions>*> sessionStores; //pointers remain stable, thanks to std::map<> + + globalSessionStore_.access([&](GlobalSshSessions& sessionsById) + { + for (auto& [sessionId, idleSession] : sessionsById) + sessionStores.push_back(&idleSession); + }); + + for (Protected<IdleSshSessions>* sessionStore : sessionStores) + for (bool done = false; !done;) + sessionStore->access([&](IdleSshSessions& sessions) + { + for (std::unique_ptr<SshSession>& sshSession : sessions.idleSshSessions) + if (!sshSession->isHealthy()) //!isHealthy() sessions are destroyed after use => in this context this means they have been idle for too long + { + sshSession.swap(sessions.idleSshSessions.back()); + /**/ sessions.idleSshSessions.pop_back(); //run ~SshSession *inside* the lock! => avoid hitting server limits! + std::this_thread::yield(); + return; //don't hold lock for too long: delete only one session at a time, then yield... + } + eraseIf(sessions.sshSessionsWithThreadAffinity, [](const auto& v) { return !v.second.lock(); }); //clean up dangling weak pointer + done = true; + }); + } + } + + struct IdleSshSessions + { + std::vector<std::unique_ptr<SshSession>> idleSshSessions; //extract *temporarily* from this list during use + std::map<uint64_t, std::weak_ptr<SshSessionShared>> sshSessionsWithThreadAffinity; //Win32 thread IDs may be REUSED! still, shouldn't be a problem... + }; + + using GlobalSshSessions = std::map<SshSessionId, Protected<IdleSshSessions>>; + + Protected<GlobalSshSessions> globalSessionStore_; + InterruptibleThread sessionCleaner_; +}; + +//-------------------------------------------------------------------------------------- +UniInitializer globalStartupInitSftp(*globalSftpSessionCount.get()); //static ordering: place *before* SftpSessionManager instance! + +Global<SftpSessionManager> globalSftpSessionManager(std::make_unique<SftpSessionManager>()); +//-------------------------------------------------------------------------------------- + + +void SftpSessionManager::ReUseOnDelete::operator()(SshSession* s) const +{ + //assert(s); -> custom deleter is only called on non-null pointer + if (s->isHealthy()) //thread that created the "!isHealthy()" session is responsible for clean up (avoid hitting server connection limits!) + if (std::shared_ptr<SftpSessionManager> mgr = globalSftpSessionManager.get()) + { + Protected<IdleSshSessions>& sessionStore = mgr->getSessionStore(s->getSessionId()); + sessionStore.access([&](IdleSshSessions& sessions) + { + sessions.idleSshSessions.emplace_back(s); //pass ownership + }); + return; + } + + delete s; +} + + +std::shared_ptr<SftpSessionManager::SshSessionShared> getSharedSftpSession(const SftpLoginInfo& login) //throw FileError +{ + if (const std::shared_ptr<SftpSessionManager> mgr = globalSftpSessionManager.get()) + return mgr->getSharedSession(login); //throw FileError + + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(login.server)), L"Function call not allowed during process shutdown."); +} + + +std::unique_ptr<SftpSessionManager::SshSessionExclusive> getExclusiveSftpSession(const SftpLoginInfo& login) //throw FileError +{ + if (const std::shared_ptr<SftpSessionManager> mgr = globalSftpSessionManager.get()) + return mgr->getExclusiveSession(login); //throw FileError + + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(login.server)), L"Function call not allowed during process shutdown."); +} + + +void runSftpCommand(const SftpLoginInfo& login, const std::wstring& functionName, + const std::function<int(const SshSession::Details& sd)>& sftpCommand /*noexcept!*/) //throw SysError, FileError +{ + std::shared_ptr<SftpSessionManager::SshSessionShared> asyncSession = getSharedSftpSession(login); //throw FileError + //no need to protect against concurrency: shared session is (temporarily) bound to current thread + try + { + asyncSession->executeBlocking(functionName, sftpCommand); //throw SysError, FatalSshError + } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => we stop using session => map to FileError is okay +} + +//=========================================================================================================================== +//=========================================================================================================================== +struct SftpItemDetails +{ + AFS::ItemType type; + uint64_t fileSize; + time_t modTime; +}; +struct SftpItem +{ + Zstring itemName; + SftpItemDetails details; +}; +std::vector<SftpItem> getDirContentFlat(const SftpLoginInfo& login, const AfsPath& dirPath) //throw FileError +{ + LIBSSH2_SFTP_HANDLE* dirHandle = nullptr; + try + { + runSftpCommand(login, L"libssh2_sftp_opendir", //throw SysError, FileError + [&](const SshSession::Details& sd) //noexcept! + { + dirHandle = ::libssh2_sftp_opendir(sd.sftpChannel, getLibssh2Path(dirPath).c_str()); + if (!dirHandle) + return std::min(::libssh2_session_last_errno(sd.sshSession), LIBSSH2_ERROR_SOCKET_NONE); + return LIBSSH2_ERROR_NONE; + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot open directory %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, dirPath))), e.toString()); } + + ZEN_ON_SCOPE_EXIT(try + { + runSftpCommand(login, L"libssh2_sftp_closedir", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_closedir(dirHandle); }); //noexcept! + } + catch (SysError&) {} + catch (FileError&) {}); + + std::vector<char> buffer(10000); //libssh2 sample code uses 512 + std::vector<SftpItem> output; + for (;;) + { + LIBSSH2_SFTP_ATTRIBUTES attribs = {}; + int rc = 0; + try + { + runSftpCommand(login, L"libssh2_sftp_readdir", //throw SysError, FileError + [&](const SshSession::Details& sd) { return rc = ::libssh2_sftp_readdir(dirHandle, &buffer[0], buffer.size(), &attribs); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read directory %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, dirPath))), e.toString()); } + + if (rc == 0) //no more items + return output; + + const std::string sftpItemName(&buffer[0], rc); + + if (sftpItemName == "." || sftpItemName == "..") //check needed for SFTP, too! + continue; + + const Zstring& itemName = utfTo<Zstring>(sftpItemName); + const AfsPath itemPath(nativeAppendPaths(dirPath.value, itemName)); + + if ((attribs.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) == 0) //server probably does not support these attributes => fail at folder level + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, itemPath))), L"File attributes not available."); + + if (LIBSSH2_SFTP_S_ISLNK(attribs.permissions)) + { + if ((attribs.flags & LIBSSH2_SFTP_ATTR_ACMODTIME) == 0) //server probably does not support these attributes => fail at folder level + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, itemPath))), L"Modification time not supported."); + output.push_back({ itemName, { AFS::ItemType::SYMLINK, 0, static_cast<time_t>(attribs.mtime) }}); + } + else if (LIBSSH2_SFTP_S_ISDIR(attribs.permissions)) + output.push_back({ itemName, { AFS::ItemType::FOLDER, 0, static_cast<time_t>(attribs.mtime) }}); + else //a file or named pipe, ect: LIBSSH2_SFTP_S_ISREG, LIBSSH2_SFTP_S_ISCHR, LIBSSH2_SFTP_S_ISBLK, LIBSSH2_SFTP_S_ISFIFO, LIBSSH2_SFTP_S_ISSOCK + { + if ((attribs.flags & LIBSSH2_SFTP_ATTR_ACMODTIME) == 0) //server probably does not support these attributes => fail at folder level + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, itemPath))), L"Modification time not supported."); + if ((attribs.flags & LIBSSH2_SFTP_ATTR_SIZE) == 0) + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, itemPath))), L"File size not supported."); + output.push_back({ itemName, { AFS::ItemType::FILE, attribs.filesize, static_cast<time_t>(attribs.mtime) }}); + } + } +} + + +SftpItemDetails getSymlinkTargetDetails(const SftpLoginInfo& login, const AfsPath& linkPath) //throw FileError +{ + LIBSSH2_SFTP_ATTRIBUTES attribsTrg = {}; + try + { + runSftpCommand(login, L"libssh2_sftp_stat", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_stat(sd.sftpChannel, getLibssh2Path(linkPath).c_str(), &attribsTrg); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot resolve symbolic link %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, linkPath))), e.toString()); } + + if ((attribsTrg.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) == 0) + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, linkPath))), L"File attributes not available."); + + if (LIBSSH2_SFTP_S_ISDIR(attribsTrg.permissions)) + return { AFS::ItemType::FOLDER, 0, static_cast<time_t>(attribsTrg.mtime) }; + else + { + if ((attribsTrg.flags & LIBSSH2_SFTP_ATTR_ACMODTIME) == 0) //server probably does not support these attributes => should fail at folder level! + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, linkPath))), L"Modification time not supported."); + if ((attribsTrg.flags & LIBSSH2_SFTP_ATTR_SIZE) == 0) + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getSftpDisplayPath(login.server, linkPath))), L"File size not supported."); + + return { AFS::ItemType::FILE, attribsTrg.filesize, static_cast<time_t>(attribsTrg.mtime) }; + } +} + + +class SingleFolderTraverser +{ +public: + SingleFolderTraverser(const SftpLoginInfo& login, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/) : + workload_(workload), login_(login) + { + while (!workload_.empty()) + { + auto wi = std::move(workload_. back()); //yes, no strong exception guarantee (std::bad_alloc) + /**/ workload_.pop_back(); // + const auto& [folderPath, cb] = wi; + + tryReportingDirError([&] //throw X + { + traverseWithException(folderPath, *cb); //throw FileError, X + }, *cb); + } + } + +private: + SingleFolderTraverser (const SingleFolderTraverser&) = delete; + SingleFolderTraverser& operator=(const SingleFolderTraverser&) = delete; + + void traverseWithException(const AfsPath& dirPath, AFS::TraverserCallback& cb) //throw FileError, X + { + for (const SftpItem& item : getDirContentFlat(login_, dirPath)) //throw FileError + { + const AfsPath itemPath(nativeAppendPaths(dirPath.value, item.itemName)); + + switch (item.details.type) + { + case AFS::ItemType::FILE: + cb.onFile({ item.itemName, item.details.fileSize, item.details.modTime, AFS::FileId(), nullptr /*symlinkInfo*/ }); //throw X + break; + + case AFS::ItemType::FOLDER: + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ item.itemName, nullptr /*symlinkInfo*/ })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + break; + + case AFS::ItemType::SYMLINK: + switch (cb.onSymlink({ item.itemName, item.details.modTime })) //throw X + { + case AFS::TraverserCallback::LINK_FOLLOW: + { + SftpItemDetails targetDetails = {}; + if (!tryReportingItemError([&] //throw X + { + targetDetails = getSymlinkTargetDetails(login_, itemPath); //throw FileError + }, cb, item.itemName)) + continue; + + const AFS::SymlinkInfo linkInfo = { item.itemName, targetDetails.modTime }; + + if (targetDetails.type == AFS::ItemType::FOLDER) + { + if (std::shared_ptr<AFS::TraverserCallback> cbSub = cb.onFolder({ item.itemName, &linkInfo })) //throw X + workload_.push_back({ itemPath, std::move(cbSub) }); + } + else //a file or named pipe, etc. + cb.onFile({ item.itemName, targetDetails.fileSize, targetDetails.modTime, AFS::FileId(), &linkInfo }); //throw X + } + break; + + case AFS::TraverserCallback::LINK_SKIP: + break; + } + break; + } + } + } + + std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>> workload_; + const SftpLoginInfo login_; +}; + + +void traverseFolderRecursiveSftp(const SftpLoginInfo& login, const std::vector<std::pair<AfsPath, std::shared_ptr<AFS::TraverserCallback>>>& workload /*throw X*/, size_t) //throw X +{ + SingleFolderTraverser dummy(login, workload); //throw X +} + +//=========================================================================================================================== + +struct InputStreamSftp : public AbstractFileSystem::InputStream +{ + InputStreamSftp(const SftpLoginInfo& login, const AfsPath& filePath, const IOCallback& notifyUnbufferedIO /*throw X*/) : //throw FileError + session_(getSharedSftpSession(login)), //throw FileError + displayPath_(getSftpDisplayPath(login.server, filePath)), + notifyUnbufferedIO_(notifyUnbufferedIO) + { + try + { + session_->executeBlocking(L"libssh2_sftp_open", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) //noexcept! + { + fileHandle_ = ::libssh2_sftp_open(sd.sftpChannel, getLibssh2Path(filePath).c_str(), LIBSSH2_FXF_READ, 0); + if (!fileHandle_) + return std::min(::libssh2_session_last_errno(sd.sshSession), LIBSSH2_ERROR_SOCKET_NONE); + return LIBSSH2_ERROR_NONE; + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot open file %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => stop using session => map to FileError is okay + } + + ~InputStreamSftp() + { + try + { + session_->executeBlocking(L"libssh2_sftp_close", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_close(fileHandle_); }); //noexcept! + } + catch (const SysError&) {} + catch (const FatalSshError&) {} //SSH session corrupted! => stop using session + } + + size_t read(void* buffer, size_t bytesToRead) override //throw FileError, (ErrorFileLocked), X; return "bytesToRead" bytes unless end of stream! + { + const size_t blockSize = getBlockSize(); + assert(memBuf_.size() >= blockSize); + assert(bufPos_ <= bufPosEnd_ && bufPosEnd_ <= memBuf_.size()); + + auto it = static_cast<std::byte*>(buffer); + const auto itEnd = it + bytesToRead; + for (;;) + { + const size_t junkSize = std::min(static_cast<size_t>(itEnd - it), bufPosEnd_ - bufPos_); + std::memcpy(it, &memBuf_[0] + bufPos_, junkSize); + bufPos_ += junkSize; + it += junkSize; + + if (it == itEnd) + break; + //-------------------------------------------------------------------- + const size_t bytesRead = tryRead(&memBuf_[0], blockSize); //throw FileError; may return short, only 0 means EOF! => CONTRACT: bytesToRead > 0 + bufPos_ = 0; + bufPosEnd_ = bytesRead; + + if (notifyUnbufferedIO_) notifyUnbufferedIO_(bytesRead); //throw X + + if (bytesRead == 0) //end of file + break; + } + return it - static_cast<std::byte*>(buffer); + } + + size_t getBlockSize() const override { return SFTP_OPTIMAL_BLOCK_SIZE_READ; } //non-zero block size is AFS contract! + + std::optional<AFS::StreamAttributes> getAttributesBuffered() override //throw FileError + { + return {}; //although have an SFTP stream handle, attribute access requires an extra (expensive) round-trip! + //PERF: test case 148 files, 1MB: overall copy time increases by 20% if libssh2_sftp_fstat() gets called per each file + } + +private: + size_t tryRead(void* buffer, size_t bytesToRead) //throw FileError; may return short, only 0 means EOF! => CONTRACT: bytesToRead > 0 + { + //libssh2_sftp_read has same semantics as Posix read: + if (bytesToRead == 0) //"read() with a count of 0 returns zero" => indistinguishable from end of file! => check! + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + assert(bytesToRead == getBlockSize()); + + ssize_t bytesRead = 0; + try + { + session_->executeBlocking(L"libssh2_sftp_read", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) //noexcept! + { + bytesRead = ::libssh2_sftp_read(fileHandle_, static_cast<char*>(buffer), bytesToRead); + return static_cast<int>(bytesRead); + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => caller (will/should) stop using session => map to FileError is okay + + if (static_cast<size_t>(bytesRead) > bytesToRead) //better safe than sorry + throw FileError(replaceCpy(_("Cannot read file %x."), L"%x", fmtPath(displayPath_)), L"libssh2_sftp_read: buffer overflow."); //user should never see this + + return bytesRead; //"zero indicates end of file" + } + + std::shared_ptr<SftpSessionManager::SshSessionShared> session_; + const std::wstring displayPath_; + LIBSSH2_SFTP_HANDLE* fileHandle_ = nullptr; + const IOCallback notifyUnbufferedIO_; //throw X + + std::vector<std::byte> memBuf_ = std::vector<std::byte>(getBlockSize()); + size_t bufPos_ = 0; //buffered I/O; see file_io.cpp + size_t bufPosEnd_ = 0; // +}; + +//=========================================================================================================================== + +//libssh2_sftp_open fails with generic LIBSSH2_FX_FAILURE if already existing +struct OutputStreamSftp : public AbstractFileSystem::OutputStreamImpl +{ + OutputStreamSftp(const SftpLoginInfo& login, //throw FileError + const AfsPath& filePath, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) : + filePath_(filePath), + displayPath_(getSftpDisplayPath(login.server, filePath)), + session_(getSharedSftpSession(login)), //throw FileError + modTime_(modTime), + notifyUnbufferedIO_(notifyUnbufferedIO) + { + try + { + session_->executeBlocking(L"libssh2_sftp_open", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) //noexcept! + { + fileHandle_ = ::libssh2_sftp_open(sd.sftpChannel, getLibssh2Path(filePath).c_str(), + LIBSSH2_FXF_WRITE | LIBSSH2_FXF_CREAT | LIBSSH2_FXF_EXCL, + LIBSSH2_SFTP_S_IRUSR | LIBSSH2_SFTP_S_IWUSR | // + LIBSSH2_SFTP_S_IRGRP | LIBSSH2_SFTP_S_IWGRP | //0666 + LIBSSH2_SFTP_S_IROTH | LIBSSH2_SFTP_S_IWOTH); // + if (!fileHandle_) + return std::min(::libssh2_session_last_errno(sd.sshSession), LIBSSH2_ERROR_SOCKET_NONE); + return LIBSSH2_ERROR_NONE; + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => stop using session => map to FileError is okay + + //NOTE: fileHandle_ still unowned until end of constructor!!! + + //pre-allocate file space? not supported + } + + ~OutputStreamSftp() + { + if (fileHandle_) //basic exception guarantee only! in general we expect a call to finalize()! + try + { + close(); //throw FileError + } + catch (FileError&) {} + } + + void write(const void* buffer, size_t bytesToWrite) override //throw FileError, X + { + const size_t blockSize = getBlockSize(); + assert(memBuf_.size() >= blockSize); + assert(bufPos_ <= bufPosEnd_ && bufPosEnd_ <= memBuf_.size()); + + auto it = static_cast<const std::byte*>(buffer); + const auto itEnd = it + bytesToWrite; + for (;;) + { + if (memBuf_.size() - bufPos_ < blockSize) //support memBuf_.size() > blockSize to reduce memmove()s, but perf test shows: not really needed! + // || bufPos_ == bufPosEnd_) -> not needed while memBuf_.size() == blockSize + { + std::memmove(&memBuf_[0], &memBuf_[0] + bufPos_, bufPosEnd_ - bufPos_); + bufPosEnd_ -= bufPos_; + bufPos_ = 0; + } + + const size_t junkSize = std::min(static_cast<size_t>(itEnd - it), blockSize - (bufPosEnd_ - bufPos_)); + std::memcpy(&memBuf_[0] + bufPosEnd_, it, junkSize); + bufPosEnd_ += junkSize; + it += junkSize; + + if (it == itEnd) + return; + //-------------------------------------------------------------------- + const size_t bytesWritten = tryWrite(&memBuf_[bufPos_], blockSize); //throw FileError; may return short! CONTRACT: bytesToWrite > 0 + bufPos_ += bytesWritten; + if (notifyUnbufferedIO_) notifyUnbufferedIO_(bytesWritten); //throw X! + } + } + + AFS::FinalizeResult finalize() override //throw FileError, X + { + assert(bufPosEnd_ - bufPos_ <= getBlockSize()); + assert(bufPos_ <= bufPosEnd_ && bufPosEnd_ <= memBuf_.size()); + while (bufPos_ != bufPosEnd_) + { + const size_t bytesWritten = tryWrite(&memBuf_[bufPos_], bufPosEnd_ - bufPos_); //throw FileError; may return short! CONTRACT: bytesToWrite > 0 + bufPos_ += bytesWritten; + if (notifyUnbufferedIO_) notifyUnbufferedIO_(bytesWritten); //throw X! + } + + //~OutputStreamSftp() would call this one, too, but we want to propagate errors if any: + close(); //throw FileError + + //it seems libssh2_sftp_fsetstat() triggers bugs on synology server => set mtime by path! https://freefilesync.org/forum/viewtopic.php?t=1281 + + AFS::FinalizeResult result; + //result.fileId = ... -> not supported by SFTP + try + { + setModTimeIfAvailable(); //throw FileError, follows symlinks + /* is setting modtime after closing the file handle a pessimization? + SFTP: no, needed for functional correctness (synology server), just as for Native */ + } + catch (const FileError& e) { result.errorModTime = FileError(e.toString()); /*avoid slicing*/ } + + return result; + } + +private: + size_t getBlockSize() const { return SFTP_OPTIMAL_BLOCK_SIZE_WRITE; } //non-zero block size is AFS contract! + + void close() //throw FileError + { + if (!fileHandle_) + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(displayPath_)), L"Contract error: close() called more than once."); + ZEN_ON_SCOPE_EXIT(fileHandle_ = nullptr); + + try + { + session_->executeBlocking(L"libssh2_sftp_close", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_close(fileHandle_); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => caller (will/should) stop using session => map to FileError is okay + } + + size_t tryWrite(const void* buffer, size_t bytesToWrite) //throw FileError; may return short! CONTRACT: bytesToWrite > 0 + { + if (bytesToWrite == 0) + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + assert(bytesToWrite <= getBlockSize()); + + ssize_t bytesWritten = 0; + try + { + session_->executeBlocking(L"libssh2_sftp_write", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) //noexcept! + { + bytesWritten = ::libssh2_sftp_write(fileHandle_, static_cast<const char*>(buffer), bytesToWrite); + return static_cast<int>(bytesWritten); + }); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => caller (will/should) stop using session => map to FileError is okay + + if (bytesWritten > static_cast<ssize_t>(bytesToWrite)) //better safe than sorry + throw FileError(replaceCpy(_("Cannot write file %x."), L"%x", fmtPath(displayPath_)), L"libssh2_sftp_write: buffer overflow."); + + //bytesWritten == 0 is no error according to doc! + return bytesWritten; + } + + void setModTimeIfAvailable() const //throw FileError, follows symlinks + { + assert(!fileHandle_); + if (modTime_) + { + LIBSSH2_SFTP_ATTRIBUTES attribNew = {}; + attribNew.flags = LIBSSH2_SFTP_ATTR_ACMODTIME; + attribNew.mtime = static_cast<decltype(attribNew.mtime)>(*modTime_); //32-bit target! loss of data! + attribNew.atime = static_cast<decltype(attribNew.atime)>(::time(nullptr)); // + + try + { + session_->executeBlocking(L"libssh2_sftp_setstat", //throw SysError, FatalSshError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_setstat(sd.sftpChannel, getLibssh2Path(filePath_).c_str(), &attribNew); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot write modification time of %x."), L"%x", fmtPath(displayPath_)), e.toString()); } + catch (const FatalSshError& e) { throw FileError(e.toString()); } //SSH session corrupted! => caller (will/should) stop using session => map to FileError is okay + } + } + + const AfsPath filePath_; + const std::wstring displayPath_; + std::shared_ptr<SftpSessionManager::SshSessionShared> session_; + LIBSSH2_SFTP_HANDLE* fileHandle_ = nullptr; + const std::optional<time_t> modTime_; + const IOCallback notifyUnbufferedIO_; //throw X + + std::vector<std::byte> memBuf_ = std::vector<std::byte>(getBlockSize()); + size_t bufPos_ = 0; //buffered I/O see file_io.cpp + size_t bufPosEnd_ = 0; // +}; + +//=========================================================================================================================== + +class SftpFileSystem : public AbstractFileSystem +{ +public: + SftpFileSystem(const SftpLoginInfo& login) : login_(login) {} + + AfsPath getHomePath() const //throw FileError + { + try + { + //we never ever change the SFTP working directory, right? ...right? + return getServerRealPath("."); //throw SysError, FileError + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot determine final path for %x."), L"%x", fmtPath(getDisplayPath(AfsPath(Zstr("."))))), e.toString()); } + } + +private: + Zstring getInitPathPhrase(const AfsPath& afsPath) const override { return concatenateSftpFolderPathPhrase(login_, afsPath); } + + std::wstring getDisplayPath(const AfsPath& afsPath) const override { return getSftpDisplayPath(login_.server, afsPath); } + + bool isNullFileSystem() const override { return login_.server.empty(); } + + int compareDeviceSameAfsType(const AbstractFileSystem& afsRhs) const override + { + const SftpLoginInfo& lhs = login_; + const SftpLoginInfo& rhs = static_cast<const SftpFileSystem&>(afsRhs).login_; + + //exactly the type of case insensitive comparison we need for server names! + const int rv = compareAsciiNoCase(lhs.server, rhs.server); //https://msdn.microsoft.com/en-us/library/windows/desktop/ms738519#IDNs + if (rv != 0) + return rv; + + //port does NOT create a *different* data source!!! -> same thing for password! + + //consider username: different users may have different views and folder access rights! + + return compareString(lhs.username, rhs.username); //case sensitive! + } + + //---------------------------------------------------------------------------------------------------------------- + ItemType getItemType(const AfsPath& afsPath) const override //throw FileError + { + try + { + LIBSSH2_SFTP_ATTRIBUTES attr = {}; + runSftpCommand(login_, L"libssh2_sftp_lstat", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_lstat(sd.sftpChannel, getLibssh2Path(afsPath).c_str(), &attr); }); //noexcept! + + if ((attr.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) == 0) + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getDisplayPath(afsPath))), L"File attributes not available."); + + if (LIBSSH2_SFTP_S_ISLNK(attr.permissions)) + return ItemType::SYMLINK; + if (LIBSSH2_SFTP_S_ISDIR(attr.permissions)) + return ItemType::FOLDER; + return ItemType::FILE; //LIBSSH2_SFTP_S_ISREG || LIBSSH2_SFTP_S_ISCHR || LIBSSH2_SFTP_S_ISBLK || LIBSSH2_SFTP_S_ISFIFO || LIBSSH2_SFTP_S_ISSOCK + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot read file attributes of %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + std::optional<ItemType> itemStillExists(const AfsPath& afsPath) const override //throw FileError + { + //default implementation: folder traversal + return AbstractFileSystem::itemStillExists(afsPath); //throw FileError + } + //---------------------------------------------------------------------------------------------------------------- + + //already existing: fail/ignore + //=> SFTP will fail with obscure LIBSSH2_FX_FAILURE error message + void createFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + runSftpCommand(login_, L"libssh2_sftp_mkdir", //throw SysError, FileError + [&](const SshSession::Details& sd) //noexcept! + { + //- 0777, default for newly created directories + //- use LIBSSH2_SFTP_DEFAULT_MODE? bugz! https://github.com/libssh2/libssh2/pull/284 + //- fails if folder is already existing + return ::libssh2_sftp_mkdir(sd.sftpChannel, getLibssh2Path(afsPath).c_str(), LIBSSH2_SFTP_S_IRWXU | LIBSSH2_SFTP_S_IRWXG | LIBSSH2_SFTP_S_IRWXO); + }); + } + catch (const SysError& e) //libssh2_sftp_mkdir reports generic LIBSSH2_FX_FAILURE if existing + { + throw FileError(replaceCpy(_("Cannot create directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeFilePlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + runSftpCommand(login_, L"libssh2_sftp_unlink", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_unlink(sd.sftpChannel, getLibssh2Path(afsPath).c_str()); }); //noexcept! + } + catch (const SysError& e) + { + throw FileError(replaceCpy(_("Cannot delete file %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeSymlinkPlain(const AfsPath& afsPath) const override //throw FileError + { + this->removeFilePlain(afsPath); //throw FileError + } + + void removeFolderPlain(const AfsPath& afsPath) const override //throw FileError + { + try + { + runSftpCommand(login_, L"libssh2_sftp_rmdir", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_rmdir(sd.sftpChannel, getLibssh2Path(afsPath).c_str()); }); //noexcept! + } + catch (const SysError& e) + { + //tested: libssh2_sftp_rmdir will fail for symlinks! + bool symlinkExists = false; + try { symlinkExists = getItemType(afsPath) == ItemType::SYMLINK; } /*throw FileError*/ catch (FileError&) {} //previous exception is more relevant + + if (symlinkExists) + return removeSymlinkPlain(afsPath); //throw FileError + + throw FileError(replaceCpy(_("Cannot delete directory %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); + } + } + + void removeFolderIfExistsRecursion(const AfsPath& afsPath, //throw FileError + const std::function<void (const std::wstring& displayPath)>& onBeforeFileDeletion /*throw X*/, //optional + const std::function<void (const std::wstring& displayPath)>& onBeforeFolderDeletion) const override //one call for each object! + { + //default implementation: folder traversal + AbstractFileSystem::removeFolderIfExistsRecursion(afsPath, onBeforeFileDeletion, onBeforeFolderDeletion); //throw FileError, X + } + + //---------------------------------------------------------------------------------------------------------------- + AfsPath getServerRealPath(const std::string& sftpPath) const //throw SysError, FileError + { + std::vector<char> buffer(10000); + runSftpCommand(login_, L"libssh2_sftp_realpath", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_realpath(sd.sftpChannel, sftpPath.c_str(), &buffer[0], static_cast<unsigned int>(buffer.size())); }); //noexcept! + + const std::string sftpPathTrg = &buffer[0]; + if (!startsWith(sftpPathTrg, '/')) + throw SysError(replaceCpy<std::wstring>(L"Invalid path %x.", L"%x", fmtPath(utfTo<std::wstring>(sftpPathTrg)))); + + return sanitizeRootRelativePath(utfTo<Zstring>(sftpPathTrg)); //code-reuse! but the sanitize part isn't really needed here... + } + + AbstractPath getSymlinkResolvedPath(const AfsPath& afsPath) const override //throw FileError + { + try + { + const AfsPath afsPathTrg = getServerRealPath(getLibssh2Path(afsPath)); //throw SysError, FileError + return AbstractPath(makeSharedRef<SftpFileSystem>(login_), afsPathTrg); + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot determine final path for %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + } + + std::string getSymlinkBinaryContent(const AfsPath& afsPath) const override //throw FileError + { + std::vector<char> buffer(10000); + try + { + runSftpCommand(login_, L"libssh2_sftp_readlink", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_readlink(sd.sftpChannel, getLibssh2Path(afsPath).c_str(), &buffer[0], static_cast<unsigned int>(buffer.size())); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot resolve symbolic link %x."), L"%x", fmtPath(getDisplayPath(afsPath))), e.toString()); } + + return &buffer[0]; + } + //---------------------------------------------------------------------------------------------------------------- + + //return value always bound: + std::unique_ptr<InputStream> getInputStream(const AfsPath& afsPath, const IOCallback& notifyUnbufferedIO /*throw X*/) const override //throw FileError, (ErrorFileLocked) + { + return std::make_unique<InputStreamSftp>(login_, afsPath, notifyUnbufferedIO); //throw FileError + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) => SFTP will fail with obscure LIBSSH2_FX_FAILURE error message + std::unique_ptr<OutputStreamImpl> getOutputStream(const AfsPath& afsPath, //throw FileError + std::optional<uint64_t> streamSize, + std::optional<time_t> modTime, + const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + return std::make_unique<OutputStreamSftp>(login_, afsPath, modTime, notifyUnbufferedIO); //throw FileError + } + + //---------------------------------------------------------------------------------------------------------------- + void traverseFolderRecursive(const TraverserWorkload& workload /*throw X*/, size_t parallelOps) const override + { + traverseFolderRecursiveSftp(login_, workload /*throw X*/, parallelOps); //throw X + } + //---------------------------------------------------------------------------------------------------------------- + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + //symlink handling: follow link! + FileCopyResult copyFileForSameAfsType(const AfsPath& afsPathSource, const StreamAttributes& attrSource, //throw FileError, (ErrorFileLocked), X + const AbstractPath& apTarget, bool copyFilePermissions, const IOCallback& notifyUnbufferedIO /*throw X*/) const override + { + //no native SFTP file copy => use stream-based file copy: + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for SFTP devices."); + + //target existing: undefined behavior! (fail/overwrite/auto-rename) + return copyFileAsStream(afsPathSource, attrSource, apTarget, notifyUnbufferedIO); //throw FileError, (ErrorFileLocked), X + } + + //already existing: fail/ignore + //symlink handling: follow link! + void copyNewFolderForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override //throw FileError + { + if (copyFilePermissions) + throw FileError(replaceCpy(_("Cannot write permissions of %x."), L"%x", fmtPath(AFS::getDisplayPath(apTarget))), + L"Permissions not supported for SFTP devices."); + + //already existing: fail/ignore + AFS::createFolderPlain(apTarget); //throw FileError + } + + void copySymlinkForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget, bool copyFilePermissions) const override + { + throw FileError(replaceCpy(replaceCpy(_("Cannot copy symbolic link %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))), + L"Setting symlink modtime not supported by libssh2."); + } + + //target existing: undefined behavior! (fail/overwrite/auto-rename) => SFTP will fail with obscure LIBSSH2_FX_FAILURE error message + void moveAndRenameItemForSameAfsType(const AfsPath& afsPathSource, const AbstractPath& apTarget) const override //throw FileError, ErrorDifferentVolume + { + auto generateErrorMsg = [&] { return replaceCpy(replaceCpy(_("Cannot move file %x to %y."), + L"%x", L"\n" + fmtPath(getDisplayPath(afsPathSource))), + L"%y", L"\n" + fmtPath(AFS::getDisplayPath(apTarget))); + }; + + if (compareDeviceSameAfsType(apTarget.afsDevice.ref()) != 0) + throw ErrorDifferentVolume(generateErrorMsg(), L"Different SFTP volume."); + + try + { + runSftpCommand(login_, L"libssh2_sftp_rename_ex", //throw SysError, FileError + [&](const SshSession::Details& sd) //noexcept! + { + /* + LIBSSH2_SFTP_RENAME_NATIVE: "The server is free to do the rename operation in whatever way it chooses. Any other set flags are to be taken as hints to the server." No, thanks! + + LIBSSH2_SFTP_RENAME_OVERWRITE: "No overwriting rename in [SFTP] v3/v4" http://www.greenend.org.uk/rjk/sftp/sftpversions.html + Test: LIBSSH2_SFTP_RENAME_OVERWRITE is not honored on freefilesync.org, no matter if LIBSSH2_SFTP_RENAME_NATIVE is set or not + => makes sense since SFTP v3 does not honor the additional flags that libssh2 sends! + + "... the most widespread SFTP server implementation, the OpenSSH, will fail the SSH_FXP_RENAME request if the target file already exists" + => incidentally this is just the behavior we want! + */ + const std::string sftpPathOld = getLibssh2Path(afsPathSource); + const std::string sftpPathNew = getLibssh2Path(apTarget.afsPath); + + return ::libssh2_sftp_rename_ex(sd.sftpChannel, + sftpPathOld.c_str(), static_cast<unsigned int>(sftpPathOld.size()), + sftpPathNew.c_str(), static_cast<unsigned int>(sftpPathNew.size()), + LIBSSH2_SFTP_RENAME_ATOMIC); + }); + } + catch (const SysError& e) //libssh2_sftp_rename_ex reports generic LIBSSH2_FX_FAILURE if target is already existing! + { + throw FileError(generateErrorMsg(), e.toString()); + } + } + + bool supportsPermissions(const AfsPath& afsPath) const override { return false; } //throw FileError + //wait until there is real demand for copying from and to SFTP with permissions => use stream-based file copy: + + //---------------------------------------------------------------------------------------------------------------- + ImageHolder getFileIcon (const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } //noexcept; optional return value + ImageHolder getThumbnailImage(const AfsPath& afsPath, int pixelSize) const override { return ImageHolder(); } // + + void authenticateAccess(bool allowUserInteraction) const override {} //throw FileError + + int getAccessTimeout() const override { return login_.timeoutSec; } //returns "0" if no timeout in force + + bool hasNativeTransactionalCopy() const override { return false; } + //---------------------------------------------------------------------------------------------------------------- + + uint64_t getFreeDiskSpace(const AfsPath& afsPath) const override //throw FileError, returns 0 if not available + { + //statvfs is an SFTP v3 extension and not supported by all server implementations + //Mikrotik SFTP server fails with LIBSSH2_FX_OP_UNSUPPORTED and corrupts session so that next SFTP call will hang + //(Server sends a duplicate SSH_FX_OP_UNSUPPORTED response with seemingly corrupt body and fails to respond from now on) + //https://freefilesync.org/forum/viewtopic.php?t=618 + //Just discarding the current session is not enough in all cases, e.g. 1. Open SFTP file handle 2. statvfs fails 3. must close file handle + return 0; +#if 0 + const std::string sftpPath = "/"; //::libssh2_sftp_statvfs will fail if path is not yet existing, OTOH root path should work, too? + //NO, for correctness we must check free space for the given folder!! + + //"It is unspecified whether all members of the returned struct have meaningful values on all file systems." + LIBSSH2_SFTP_STATVFS fsStats = {}; + try + { + runSftpCommand(login_, L"libssh2_sftp_statvfs", //throw SysError, FileError + [&](const SshSession::Details& sd) { return ::libssh2_sftp_statvfs(sd.sftpChannel, sftpPath.c_str(), sftpPath.size(), &fsStats); }); //noexcept! + } + catch (const SysError& e) { throw FileError(replaceCpy(_("Cannot determine free disk space for %x."), L"%x", fmtPath(getDisplayPath(L"/"))), e.toString()); } + + static_assert(sizeof(fsStats.f_bsize) >= 8); + return fsStats.f_bsize * fsStats.f_bavail; +#endif + } + + bool supportsRecycleBin(const AfsPath& afsPath, const std::function<void ()>& onUpdateGui) const override { return false; } //throw FileError + + std::unique_ptr<RecycleSession> createRecyclerSession(const AfsPath& afsPath) const override //throw FileError, return value must be bound! + { + assert(false); //see supportsRecycleBin() + throw FileError(L"Recycle Bin not supported for SFTP devices."); + } + + void recycleItemIfExists(const AfsPath& afsPath) const override //throw FileError + { + assert(false); //see supportsRecycleBin() + throw FileError(replaceCpy(_("Unable to move %x to the recycle bin."), L"%x", fmtPath(getDisplayPath(afsPath))), L"Recycle Bin not supported for SFTP devices."); + } + + const SftpLoginInfo login_; +}; + +//=========================================================================================================================== + +//expects "clean" login data, see condenseToSftpFolderPathPhrase() +Zstring concatenateSftpFolderPathPhrase(const SftpLoginInfo& login, const AfsPath& afsPath) //noexcept +{ + Zstring port; + if (login.port > 0) + port = Zstr(":") + numberTo<Zstring>(login.port); + + Zstring options; + if (login.traverserChannelsPerConnection > 1) + options += Zstr("|chan=") + numberTo<Zstring>(login.traverserChannelsPerConnection); + + if (login.timeoutSec != SftpLoginInfo().timeoutSec) + options += Zstr("|timeout=") + numberTo<Zstring>(login.timeoutSec); + + switch (login.authType) + { + case SftpAuthType::PASSWORD: + break; + + case SftpAuthType::KEY_FILE: + options += Zstr("|keyfile=") + login.privateKeyFilePath; + break; + + case SftpAuthType::AGENT: + options += Zstr("|agent"); + break; + } + + if (login.authType != SftpAuthType::AGENT) + if (!login.password.empty()) //password always last => visually truncated by folder input field + options += Zstr("|pass64=") + encodePasswordBase64(login.password); + + return Zstring(sftpPrefix) + Zstr("//") + encodeFtpUsername(login.username) + Zstr("@") + login.server + port + getServerRelPath(afsPath) + options; +} +} + +Zstring fff::condenseToSftpFolderPathPhrase(const SftpLoginInfo& login, const Zstring& relPath) //noexcept +{ + SftpLoginInfo loginTmp = login; + + //clean-up input: + trim(loginTmp.server); + trim(loginTmp.username); + trim(loginTmp.privateKeyFilePath); + + loginTmp.traverserChannelsPerConnection = std::max(1, loginTmp.traverserChannelsPerConnection); + loginTmp.timeoutSec = std::max(1, loginTmp.timeoutSec); + + if (startsWithAsciiNoCase(loginTmp.server, Zstr("http:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("https:")) || + startsWithAsciiNoCase(loginTmp.server, Zstr("ftp:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("ftps:" )) || + startsWithAsciiNoCase(loginTmp.server, Zstr("sftp:" ))) + loginTmp.server = afterFirst(loginTmp.server, Zstr(':'), IF_MISSING_RETURN_NONE); + trim(loginTmp.server, true, false, [](Zchar c) { return c == Zstr('/') || c == Zstr('\\'); }); + + return concatenateSftpFolderPathPhrase(loginTmp, sanitizeRootRelativePath(relPath)); +} + + +int fff::getServerMaxChannelsPerConnection(const SftpLoginInfo& login) //throw FileError +{ + const auto startTime = std::chrono::steady_clock::now(); + + std::unique_ptr<SftpSessionManager::SshSessionExclusive> exSession = getExclusiveSftpSession(login); //throw FileError + + ZEN_ON_SCOPE_EXIT(exSession->markAsCorrupted()); //after hitting the server limits, the session might have gone bananas (= server fails on all requests) + + for (;;) + { + try + { + SftpSessionManager::SshSessionExclusive::addSftpChannel({ exSession.get() }); //throw FileError, FatalSshError + } + catch (const FileError& ) { if (exSession->getSftpChannelCount() == 0) throw; return static_cast<int>(exSession->getSftpChannelCount()); } + catch (const FatalSshError& e) { if (exSession->getSftpChannelCount() == 0) throw FileError(e.toString()); return static_cast<int>(exSession->getSftpChannelCount()); } + + if (numeric::dist(std::chrono::steady_clock::now(), startTime) > SFTP_CHANNEL_LIMIT_DETECTION_TIME_OUT) + throw FileError(replaceCpy(_("Unable to connect to %x."), L"%x", fmtPath(login.server)) + L" " + + replaceCpy(_("Failed to open SFTP channel number %x."), L"%x", numberTo<std::wstring>(exSession->getSftpChannelCount() + 1)), + _P("Operation timed out after 1 second.", "Operation timed out after %x seconds.", + std::chrono::seconds(SFTP_CHANNEL_LIMIT_DETECTION_TIME_OUT).count())); + } +} + + +AfsPath fff::getSftpHomePath(const SftpLoginInfo& login) //throw FileError +{ + return SftpFileSystem(login).getHomePath(); //throw FileError +} + + +//syntax: sftp://[<user>[:<password>]@]<server>[:port]/<relative-path>[|option_name=value] +// +// e.g. sftp://user001:secretpassword@private.example.com:222/mydirectory/ +// sftp://user001@private.example.com/mydirectory|con=2|cpc=10|keyfile=%AppData%\id_rsa|pass64=c2VjcmV0cGFzc3dvcmQ +SftpPathInfo fff::getResolvedSftpPath(const Zstring& folderPathPhrase) //noexcept +{ + Zstring pathPhrase = expandMacros(folderPathPhrase); //expand before trimming! + trim(pathPhrase); + + if (startsWithAsciiNoCase(pathPhrase, sftpPrefix)) + pathPhrase = pathPhrase.c_str() + strLength(sftpPrefix); + trim(pathPhrase, true, false, [](Zchar c) { return c == Zstr('/') || c == Zstr('\\'); }); + + const Zstring credentials = beforeFirst(pathPhrase, Zstr('@'), IF_MISSING_RETURN_NONE); + const Zstring fullPathOpt = afterFirst(pathPhrase, Zstr('@'), IF_MISSING_RETURN_ALL); + + SftpLoginInfo login; + login.username = decodeFtpUsername(beforeFirst(credentials, Zstr(':'), IF_MISSING_RETURN_ALL)); //support standard FTP syntax, even though ":" + login.password = afterFirst(credentials, Zstr(':'), IF_MISSING_RETURN_NONE); //is not used by our concatenateSftpFolderPathPhrase()! + + const Zstring fullPath = beforeFirst(fullPathOpt, Zstr('|'), IF_MISSING_RETURN_ALL); + const Zstring options = afterFirst(fullPathOpt, Zstr('|'), IF_MISSING_RETURN_NONE); + + auto it = std::find_if(fullPath.begin(), fullPath.end(), [](Zchar c) { return c == '/' || c == '\\'; }); + const Zstring serverPort(fullPath.begin(), it); + const AfsPath serverRelPath = sanitizeRootRelativePath({ it, fullPath.end() }); + + login.server = beforeLast(serverPort, Zstr(':'), IF_MISSING_RETURN_ALL); + const Zstring port = afterLast(serverPort, Zstr(':'), IF_MISSING_RETURN_NONE); + login.port = stringTo<int>(port); //0 if empty + + if (!options.empty()) + { + for (const Zstring& optPhrase : split(options, Zstr("|"), SplitType::SKIP_EMPTY)) + if (startsWith(optPhrase, Zstr("chan="))) + login.traverserChannelsPerConnection = stringTo<int>(afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE)); + else if (startsWith(optPhrase, Zstr("timeout="))) + login.timeoutSec = stringTo<int>(afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE)); + else if (startsWith(optPhrase, Zstr("keyfile="))) + { + login.authType = SftpAuthType::KEY_FILE; + login.privateKeyFilePath = afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE); + } + else if (optPhrase == Zstr("agent")) + login.authType = SftpAuthType::AGENT; + else if (startsWith(optPhrase, Zstr("pass64="))) + login.password = decodePasswordBase64(afterFirst(optPhrase, Zstr("="), IF_MISSING_RETURN_NONE)); + else + assert(false); + } //fix "-Wdangling-else" + return { login, serverRelPath }; +} + + +bool fff::acceptsItemPathPhraseSftp(const Zstring& itemPathPhrase) //noexcept +{ + Zstring path = expandMacros(itemPathPhrase); //expand before trimming! + trim(path); + return startsWithAsciiNoCase(path, sftpPrefix); //check for explicit SFTP path +} + + +AbstractPath fff::createItemPathSftp(const Zstring& itemPathPhrase) //noexcept +{ + const SftpPathInfo pi = getResolvedSftpPath(itemPathPhrase); //noexcept + return AbstractPath(makeSharedRef<SftpFileSystem>(pi.login), pi.afsPath); +} diff --git a/FreeFileSync/Source/fs/sftp.h b/FreeFileSync/Source/fs/sftp.h new file mode 100644 index 00000000..dbd73556 --- /dev/null +++ b/FreeFileSync/Source/fs/sftp.h @@ -0,0 +1,57 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef SFTP_H_5392187498172458215426 +#define SFTP_H_5392187498172458215426 + +#include "abstract.h" + + +namespace fff +{ +bool acceptsItemPathPhraseSftp(const Zstring& itemPathPhrase); //noexcept +AbstractPath createItemPathSftp(const Zstring& itemPathPhrase); //noexcept + +//------------------------------------------------------- + +enum class SftpAuthType +{ + PASSWORD, + KEY_FILE, + AGENT, +}; + +struct SftpLoginInfo +{ + Zstring server; + int port = 0; // > 0 if set + Zstring username; + + SftpAuthType authType = SftpAuthType::PASSWORD; + Zstring password; //authType == PASSWORD or KEY_FILE + Zstring privateKeyFilePath; //authType == KEY_FILE: use PEM-encoded private key (protected by password) for authentication + + //other settings not specific to SFTP session: + int traverserChannelsPerConnection = 1; //valid range: [1, inf) + int timeoutSec = 15; // +}; + +struct SftpPathInfo +{ + SftpLoginInfo login; + AfsPath afsPath; //server-relative path +}; +SftpPathInfo getResolvedSftpPath(const Zstring& folderPathPhrase); //noexcept + +//expects (potentially messy) user input: +Zstring condenseToSftpFolderPathPhrase(const SftpLoginInfo& login, const Zstring& relPath); //noexcept + +int getServerMaxChannelsPerConnection(const SftpLoginInfo& login); //throw FileError + +AfsPath getSftpHomePath(const SftpLoginInfo& login); //throw FileError +} + +#endif //SFTP_H_5392187498172458215426 diff --git a/FreeFileSync/Source/ui/abstract_folder_picker.cpp b/FreeFileSync/Source/ui/abstract_folder_picker.cpp new file mode 100644 index 00000000..5b336a02 --- /dev/null +++ b/FreeFileSync/Source/ui/abstract_folder_picker.cpp @@ -0,0 +1,387 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#include "abstract_folder_picker.h" +#include <wx+/async_task.h> +#include <wx+/std_button_layout.h> +#include <wx+/image_resources.h> +#include <wx+/popup_dlg.h> +#include <wx+/image_tools.h> +#include "gui_generated.h" +#include "../base/icon_buffer.h" + +using namespace zen; +using namespace fff; +using AFS = AbstractFileSystem; + + +namespace +{ +enum class NodeLoadStatus +{ + notLoaded, + loading, + loaded +}; + +struct AfsTreeItemData : public wxTreeItemData +{ + AfsTreeItemData(const AbstractPath& path) : folderPath(path) {} + + const AbstractPath folderPath; + std::wstring errorMsg; //optional + NodeLoadStatus loadStatus = NodeLoadStatus::notLoaded; + std::vector<std::function<void()>> onLoadCompleted; //bound! +}; + + +wxString getNodeDisplayName(const AbstractPath& folderPath) +{ + if (!AFS::getParentPath(folderPath)) //server root + return utfTo<wxString>(FILE_NAME_SEPARATOR); + + return utfTo<wxString>(AFS::getItemName(folderPath)); +} + + +class AbstractFolderPickerDlg : public AbstractFolderPickerGenerated +{ +public: + AbstractFolderPickerDlg(wxWindow* parent, AbstractPath& folderPath); + +private: + void OnOkay (wxCommandEvent& event) override; + void OnCancel(wxCommandEvent& event) override { EndModal(ReturnAfsPicker::BUTTON_CANCEL); } + void OnClose (wxCloseEvent& event) override { EndModal(ReturnAfsPicker::BUTTON_CANCEL); } + + void OnKeyPressed(wxKeyEvent& event); + void OnExpandNode(wxTreeEvent& event) override; + void OnItemTooltip(wxTreeEvent& event); + + void populateNodeThen(const wxTreeItemId& itemId, const std::function<void()>& evalOnGui /*optional*/, bool popupErrors); + + void findAndNavigateToExistingPath(const AbstractPath& folderPath); + void navigateToExistingPath(const wxTreeItemId& itemId, const std::vector<Zstring>& nodeRelPath, AFS::ItemType leafType); + + enum class TreeNodeImage + { + root = 0, //used as zero-based wxImageList index! + folder, + folderSymlink, + error + }; + + AsyncGuiQueue guiQueue_{ 25 /*polling [ms]*/ }; //schedule and run long-running tasks asynchronously, but process results on GUI queue + + //output-only parameters: + AbstractPath& folderPathOut_; +}; + + +AbstractFolderPickerDlg::AbstractFolderPickerDlg(wxWindow* parent, AbstractPath& folderPath) : + AbstractFolderPickerGenerated(parent), + folderPathOut_(folderPath) +{ + setStandardButtonLayout(*bSizerStdButtons, StdButtons().setAffirmative(m_buttonOkay).setCancel(m_buttonCancel)); + + m_staticTextStatus->SetLabel(L""); + m_treeCtrlFileSystem->SetMinSize(wxSize(fastFromDIP(350), fastFromDIP(400))); + + const int iconSize = IconBuffer::getSize(IconBuffer::SIZE_SMALL); + auto imgList = std::make_unique<wxImageList>(iconSize, iconSize); + + //add images in same sequence like TreeNodeImage enum!!! + imgList->Add(shrinkImage(getResourceImage(L"server").ConvertToImage(), iconSize)); + imgList->Add( IconBuffer::genericDirIcon(IconBuffer::SIZE_SMALL)); + imgList->Add(layOver(IconBuffer::genericDirIcon(IconBuffer::SIZE_SMALL), IconBuffer::linkOverlayIcon(IconBuffer::SIZE_SMALL))); + imgList->Add(shrinkImage(getResourceImage(L"msg_error").ConvertToImage(), iconSize)); + assert(imgList->GetImageCount() == static_cast<int>(TreeNodeImage::error) + 1); + + m_treeCtrlFileSystem->AssignImageList(imgList.release()); //pass ownership + + const AbstractPath rootPath(folderPath.afsDevice, AfsPath()); + + const wxTreeItemId rootId = m_treeCtrlFileSystem->AddRoot(getNodeDisplayName(rootPath), static_cast<int>(TreeNodeImage::root), -1, + new AfsTreeItemData(rootPath)); + m_treeCtrlFileSystem->SetItemHasChildren(rootId); + + if (!AFS::getParentPath(folderPath)) //server root + populateNodeThen(rootId, [this, rootId] { m_treeCtrlFileSystem->Expand(rootId); }, true /*popupErrors*/); + else + try //folder picker has dual responsibility: + { + //1. test server connection: + const AFS::ItemType type = AFS::getItemType(folderPath); //throw FileError + //2. navigate + select path + navigateToExistingPath(rootId, split(folderPath.afsPath.value, FILE_NAME_SEPARATOR, SplitType::SKIP_EMPTY), type); + } + catch (const FileError& e) //not existing or access error + { + findAndNavigateToExistingPath(*AFS::getParentPath(folderPath)); //let's run async while the error message is shown :) + + showNotificationDialog(parent /*"this" not yet shown!*/, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e.toString())); + } + + //---------------------------------------------------------------------- + GetSizer()->SetSizeHints(this); //~=Fit() + SetMinSize() + //=> works like a charm for GTK2 with window resizing problems and title bar corruption; e.g. Debian!!! + Center(); //needs to be re-applied after a dialog size change! + + Connect(wxEVT_CHAR_HOOK, wxKeyEventHandler (AbstractFolderPickerDlg::OnKeyPressed), nullptr, this); //dialog-specific local key events + Connect(wxEVT_TREE_ITEM_GETTOOLTIP, wxTreeEventHandler(AbstractFolderPickerDlg::OnItemTooltip), nullptr, this); + + m_treeCtrlFileSystem->SetFocus(); +} + + +void AbstractFolderPickerDlg::OnKeyPressed(wxKeyEvent& event) +{ + switch (event.GetKeyCode()) + { + //wxTreeCtrl seems to eat up ENTER without adding any functionality; we can do better: + case WXK_RETURN: + case WXK_NUMPAD_ENTER: + { + wxCommandEvent dummy(wxEVT_COMMAND_BUTTON_CLICKED); + OnOkay(dummy); + return; + } + } + event.Skip(); +} + + +struct FlatTraverserCallback : public AFS::TraverserCallback +{ + struct Result + { + std::map<Zstring, bool /*is symlink*/> folderNames; + std::wstring errorMsg; + }; + + const Result& getResult() { return result_; } + +private: + void onFile (const AFS::FileInfo& fi) override {} + std::shared_ptr<TraverserCallback> onFolder (const AFS::FolderInfo& fi) override { result_.folderNames.emplace(fi.itemName, fi.symlinkInfo != nullptr); return nullptr; } + HandleLink onSymlink(const AFS::SymlinkInfo& si) override { return LINK_FOLLOW; } + HandleError reportDirError (const std::wstring& msg, size_t retryNumber) override { logError(msg); return ON_ERROR_CONTINUE; } + HandleError reportItemError(const std::wstring& msg, size_t retryNumber, const Zstring& itemName) override { logError(msg); return ON_ERROR_CONTINUE; } + + void logError(const std::wstring& msg) + { + if (result_.errorMsg.empty()) + result_.errorMsg = msg; + } + + Result result_; +}; + + +void AbstractFolderPickerDlg::populateNodeThen(const wxTreeItemId& itemId, const std::function<void()>& evalOnGui, bool popupErrors) +{ + if (auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(itemId))) + { + switch (itemData->loadStatus) + { + case NodeLoadStatus::notLoaded: + { + if (evalOnGui) + itemData->onLoadCompleted.push_back(evalOnGui); + + itemData->loadStatus = NodeLoadStatus::loading; + + m_treeCtrlFileSystem->SetItemText(itemId, getNodeDisplayName(itemData->folderPath) + L" (" + _("Loading...") + L")"); + + guiQueue_.processAsync([folderPath = itemData->folderPath] //AbstractPath is thread-safe like an int! + { + auto ft = std::make_shared<FlatTraverserCallback>(); //noexcept, traverse directory one level deep + AFS::traverseFolderRecursive(folderPath.afsDevice, {{ folderPath.afsPath, ft }}, 1 /*parallelOps*/); + return ft->getResult(); + }, + + [this, itemId, popupErrors](const FlatTraverserCallback::Result& result) + { + if (auto itemData2 = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(itemId))) + { + m_treeCtrlFileSystem->SetItemText(itemId, getNodeDisplayName(itemData2->folderPath)); //remove "loading" phrase + + if (result.folderNames.empty()) + m_treeCtrlFileSystem->SetItemHasChildren(itemId, false); + else + { + //let's not use the wxTreeCtrl::OnCompareItems() abomination to implement sorting: + std::vector<std::pair<Zstring, bool /*is symlink*/>> folderNamesSorted(result.folderNames.begin(), result.folderNames.end()); + std::sort(folderNamesSorted.begin(), folderNamesSorted.end(), [](const auto& lhs, const auto& rhs) { return LessNaturalSort()(lhs.first, rhs.first); }); + + for (const auto& [childName, isSymlink] : folderNamesSorted) + { + const AbstractPath childFolderPath = AFS::appendRelPath(itemData2->folderPath, childName); + + wxTreeItemId childId = m_treeCtrlFileSystem->AppendItem(itemId, getNodeDisplayName(childFolderPath), + static_cast<int>(isSymlink ? TreeNodeImage::folderSymlink : TreeNodeImage::folder), -1, + new AfsTreeItemData(childFolderPath)); + m_treeCtrlFileSystem->SetItemHasChildren(childId); + } + } + + if (!result.errorMsg.empty()) + { + m_treeCtrlFileSystem->SetItemImage(itemId, static_cast<int>(TreeNodeImage::error)); + itemData2->errorMsg = result.errorMsg; + + if (popupErrors) + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(result.errorMsg)); + } + + itemData2->loadStatus = NodeLoadStatus::loaded; //set status *before* running callbacks + for (const auto& evalOnGui2 : itemData2->onLoadCompleted) + evalOnGui2(); + } + }); + } + break; + + case NodeLoadStatus::loading: + if (evalOnGui) itemData->onLoadCompleted.push_back(evalOnGui); + break; + + case NodeLoadStatus::loaded: + if (evalOnGui) evalOnGui(); + break; + } + } +} + + +//1. find longest existing/accessible (parent) path +void AbstractFolderPickerDlg::findAndNavigateToExistingPath(const AbstractPath& folderPath) +{ + if (!AFS::getParentPath(folderPath)) + return m_staticTextStatus->SetLabel(L""); + + m_staticTextStatus->SetLabel(_("Scanning...") + L" " + utfTo<std::wstring>(FILE_NAME_SEPARATOR + folderPath.afsPath.value)); //keep it short! + + guiQueue_.processAsync([folderPath]() -> std::optional<AFS::ItemType> + { + try + { + return AFS::getItemType(folderPath); //throw FileError + } + catch (FileError&) { return std::nullopt; } //not existing or access error + }, + + [this, folderPath](std::optional<AFS::ItemType> type) + { + if (type) + { + m_staticTextStatus->SetLabel(L""); + navigateToExistingPath(m_treeCtrlFileSystem->GetRootItem(), split(folderPath.afsPath.value, FILE_NAME_SEPARATOR, SplitType::SKIP_EMPTY), *type); + } + else //split into multiple small async tasks rather than a single large one! + findAndNavigateToExistingPath(*AFS::getParentPath(folderPath)); + }); +} + + +//2. navgiate while ignoring any intermediate (access) errors or problems with hidden folders +void AbstractFolderPickerDlg::navigateToExistingPath(const wxTreeItemId& itemId, const std::vector<Zstring>& nodeRelPath, AFS::ItemType leafType) +{ + if (nodeRelPath.empty() || + (nodeRelPath.size() == 1 && leafType == AFS::ItemType::FILE)) //let's be *uber* correct + { + m_treeCtrlFileSystem->SelectItem(itemId); + //m_treeCtrlFileSystem->EnsureVisible(itemId); -> not needed: maybe wxTreeCtrl::Expand() does this? + return; + } + + populateNodeThen(itemId, [this, itemId, nodeRelPath, leafType] + { + const Zstring childFolderName = nodeRelPath.front(); + const std::vector<Zstring> childFolderRelPath{ nodeRelPath.begin() + 1, nodeRelPath.end() }; + + wxTreeItemId childIdMatch; + size_t insertPos = 0; //let's not use the wxTreeCtrl::OnCompareItems() abomination to implement sorting + + wxTreeItemIdValue cookie = nullptr; + for (wxTreeItemId childId = m_treeCtrlFileSystem->GetFirstChild(itemId, cookie); + childId.IsOk(); + childId = m_treeCtrlFileSystem->GetNextChild(itemId, cookie)) + if (auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(childId))) + { + const Zstring& itemName = AFS::getItemName(itemData->folderPath); + + if (LessNaturalSort()(itemName, childFolderName)) + ++insertPos; //assume items are already naturally sorted, see populateNodeThen() + + if (equalNoCase(itemName, childFolderName)) + { + childIdMatch = childId; + if (itemName == childFolderName) + break; //exact match => no need to search further! + } + } + + //we *know* that childFolder exists: Maybe it's just hidden during browsing: https://freefilesync.org/forum/viewtopic.php?t=3809 + if (!childIdMatch.IsOk()) // or access to root folder is denied: https://freefilesync.org/forum/viewtopic.php?t=5999 + if (auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(itemId))) + { + m_treeCtrlFileSystem->SetItemHasChildren(itemId); + + const AbstractPath childFolderPath = AFS::appendRelPath(itemData->folderPath, childFolderName); + + childIdMatch = m_treeCtrlFileSystem->InsertItem(itemId, insertPos, getNodeDisplayName(childFolderPath), + static_cast<int>(childFolderRelPath.empty() && leafType == AFS::ItemType::SYMLINK ? + TreeNodeImage::folderSymlink : TreeNodeImage::folder), -1, + new AfsTreeItemData(childFolderPath)); + m_treeCtrlFileSystem->SetItemHasChildren(childIdMatch); + } + + m_treeCtrlFileSystem->Expand(itemId); //wxTreeCtr::Expand emits wxTreeEvent!!! + + navigateToExistingPath(childIdMatch, childFolderRelPath, leafType); + }, false /*popupErrors*/); +} + + +void AbstractFolderPickerDlg::OnExpandNode(wxTreeEvent& event) +{ + const wxTreeItemId itemId = event.GetItem(); + + if (auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(itemId))) + if (itemData->loadStatus != NodeLoadStatus::loaded) + populateNodeThen(itemId, [this, itemId]() { m_treeCtrlFileSystem->Expand(itemId); }, true /*popupErrors*/); //wxTreeCtr::Expand emits wxTreeEvent!!! watch out for recursion! +} + + +void AbstractFolderPickerDlg::OnItemTooltip(wxTreeEvent& event) +{ + wxString tooltip; + if (auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(event.GetItem()))) + tooltip = itemData->errorMsg; + event.SetToolTip(tooltip); +} + + +void AbstractFolderPickerDlg::OnOkay(wxCommandEvent& event) +{ + const wxTreeItemId itemId = m_treeCtrlFileSystem->GetFocusedItem(); + + auto itemData = dynamic_cast<AfsTreeItemData*>(m_treeCtrlFileSystem->GetItemData(itemId)); + assert(itemData); + if (itemData) + folderPathOut_ = itemData->folderPath; + + EndModal(ReturnAfsPicker::BUTTON_OKAY); +} +} + + +ReturnAfsPicker::ButtonPressed fff::showAbstractFolderPicker(wxWindow* parent, AbstractPath& folderPath) +{ + AbstractFolderPickerDlg pickerDlg(parent, folderPath); + return static_cast<ReturnAfsPicker::ButtonPressed>(pickerDlg.ShowModal()); +} diff --git a/FreeFileSync/Source/ui/abstract_folder_picker.h b/FreeFileSync/Source/ui/abstract_folder_picker.h new file mode 100644 index 00000000..b8d9f1ed --- /dev/null +++ b/FreeFileSync/Source/ui/abstract_folder_picker.h @@ -0,0 +1,28 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef ABSTRACT_FOLDER_PICKER_HEADER_324872346895690 +#define ABSTRACT_FOLDER_PICKER_HEADER_324872346895690 + +#include <wx/window.h> +#include "../fs/abstract.h" + + +namespace fff +{ +struct ReturnAfsPicker +{ + enum ButtonPressed + { + BUTTON_CANCEL, + BUTTON_OKAY = 1 + }; +}; + +ReturnAfsPicker::ButtonPressed showAbstractFolderPicker(wxWindow* parent, AbstractPath& folderPath); +} + +#endif //ABSTRACT_FOLDER_PICKER_HEADER_324872346895690 diff --git a/FreeFileSync/Source/ui/app_icon.h b/FreeFileSync/Source/ui/app_icon.h index 68fa910b..68fa910b 100755..100644 --- a/FreeFileSync/Source/ui/app_icon.h +++ b/FreeFileSync/Source/ui/app_icon.h diff --git a/FreeFileSync/Source/ui/batch_config.cpp b/FreeFileSync/Source/ui/batch_config.cpp index e0b248de..e0b248de 100755..100644 --- a/FreeFileSync/Source/ui/batch_config.cpp +++ b/FreeFileSync/Source/ui/batch_config.cpp diff --git a/FreeFileSync/Source/ui/batch_config.h b/FreeFileSync/Source/ui/batch_config.h index 9a6804ca..9a6804ca 100755..100644 --- a/FreeFileSync/Source/ui/batch_config.h +++ b/FreeFileSync/Source/ui/batch_config.h diff --git a/FreeFileSync/Source/ui/batch_status_handler.cpp b/FreeFileSync/Source/ui/batch_status_handler.cpp index 345ff451..345ff451 100755..100644 --- a/FreeFileSync/Source/ui/batch_status_handler.cpp +++ b/FreeFileSync/Source/ui/batch_status_handler.cpp diff --git a/FreeFileSync/Source/ui/batch_status_handler.h b/FreeFileSync/Source/ui/batch_status_handler.h index e3a73a78..e3a73a78 100755..100644 --- a/FreeFileSync/Source/ui/batch_status_handler.h +++ b/FreeFileSync/Source/ui/batch_status_handler.h diff --git a/FreeFileSync/Source/ui/cfg_grid.cpp b/FreeFileSync/Source/ui/cfg_grid.cpp index 3d5023b4..3d5023b4 100755..100644 --- a/FreeFileSync/Source/ui/cfg_grid.cpp +++ b/FreeFileSync/Source/ui/cfg_grid.cpp diff --git a/FreeFileSync/Source/ui/cfg_grid.h b/FreeFileSync/Source/ui/cfg_grid.h index 3983df06..3983df06 100755..100644 --- a/FreeFileSync/Source/ui/cfg_grid.h +++ b/FreeFileSync/Source/ui/cfg_grid.h diff --git a/FreeFileSync/Source/ui/command_box.cpp b/FreeFileSync/Source/ui/command_box.cpp index 2ef53a51..2ef53a51 100755..100644 --- a/FreeFileSync/Source/ui/command_box.cpp +++ b/FreeFileSync/Source/ui/command_box.cpp diff --git a/FreeFileSync/Source/ui/command_box.h b/FreeFileSync/Source/ui/command_box.h index ddc6f7d0..ddc6f7d0 100755..100644 --- a/FreeFileSync/Source/ui/command_box.h +++ b/FreeFileSync/Source/ui/command_box.h diff --git a/FreeFileSync/Source/ui/file_grid.cpp b/FreeFileSync/Source/ui/file_grid.cpp index a1af13a3..52250044 100755..100644 --- a/FreeFileSync/Source/ui/file_grid.cpp +++ b/FreeFileSync/Source/ui/file_grid.cpp @@ -1586,7 +1586,7 @@ void filegrid::init(Grid& gridLeft, Grid& gridCenter, Grid& gridRight) //gridCenter.showScrollBars(Grid::SB_SHOW_NEVER, Grid::SB_SHOW_NEVER); const int widthCheckbox = getResourceImage(L"checkbox_true").GetWidth() + fastFromDIP(3); - const int widthCategory = 2 * getResourceImage(L"cat_left_only_small").GetWidth() + getResourceImage(L"notch").GetWidth(); + const int widthCategory = 2 * getResourceImage(L"cat_left_only_sicon").GetWidth() + getResourceImage(L"notch").GetWidth(); const int widthAction = 3 * getResourceImage(L"so_create_left_sicon").GetWidth(); gridCenter.SetSize(widthCategory + widthCheckbox + widthAction, -1); @@ -1779,7 +1779,7 @@ wxBitmap fff::getSyncOpImage(SyncOperation syncOp) case SO_DO_NOTHING: return getResourceImage(L"so_none_sicon"); case SO_EQUAL: - return getResourceImage(L"cat_equal_small"); + return getResourceImage(L"cat_equal_sicon"); case SO_UNRESOLVED_CONFLICT: return getResourceImage(L"cat_conflict_small"); } @@ -1793,18 +1793,18 @@ wxBitmap fff::getCmpResultImage(CompareFilesResult cmpResult) switch (cmpResult) { case FILE_LEFT_SIDE_ONLY: - return getResourceImage(L"cat_left_only_small"); + return getResourceImage(L"cat_left_only_sicon"); case FILE_RIGHT_SIDE_ONLY: - return getResourceImage(L"cat_right_only_small"); + return getResourceImage(L"cat_right_only_sicon"); case FILE_LEFT_NEWER: - return getResourceImage(L"cat_left_newer_small"); + return getResourceImage(L"cat_left_newer_sicon"); case FILE_RIGHT_NEWER: - return getResourceImage(L"cat_right_newer_small"); + return getResourceImage(L"cat_right_newer_sicon"); case FILE_DIFFERENT_CONTENT: - return getResourceImage(L"cat_different_small"); + return getResourceImage(L"cat_different_sicon"); case FILE_EQUAL: case FILE_DIFFERENT_METADATA: //= sub-category of equal - return getResourceImage(L"cat_equal_small"); + return getResourceImage(L"cat_equal_sicon"); case FILE_CONFLICT: return getResourceImage(L"cat_conflict_small"); } diff --git a/FreeFileSync/Source/ui/file_grid.h b/FreeFileSync/Source/ui/file_grid.h index 3ae06651..3ae06651 100755..100644 --- a/FreeFileSync/Source/ui/file_grid.h +++ b/FreeFileSync/Source/ui/file_grid.h diff --git a/FreeFileSync/Source/ui/file_grid_attr.h b/FreeFileSync/Source/ui/file_grid_attr.h index ea5f1b54..ea5f1b54 100755..100644 --- a/FreeFileSync/Source/ui/file_grid_attr.h +++ b/FreeFileSync/Source/ui/file_grid_attr.h diff --git a/FreeFileSync/Source/ui/file_view.cpp b/FreeFileSync/Source/ui/file_view.cpp index 47a6c178..40f0a7d5 100755..100644 --- a/FreeFileSync/Source/ui/file_view.cpp +++ b/FreeFileSync/Source/ui/file_view.cpp @@ -20,33 +20,33 @@ void addNumbers(const FileSystemObject& fsObj, StatusResult& result) visitFSObject(fsObj, [&](const FolderPair& folder) { if (!folder.isEmpty<LEFT_SIDE>()) - ++result.foldersOnLeftView; + ++result.folderCountLeft; if (!folder.isEmpty<RIGHT_SIDE>()) - ++result.foldersOnRightView; + ++result.folderCountRight; }, [&](const FilePair& file) { if (!file.isEmpty<LEFT_SIDE>()) { - result.filesizeLeftView += file.getFileSize<LEFT_SIDE>(); - ++result.filesOnLeftView; + result.bytesLeft += file.getFileSize<LEFT_SIDE>(); + ++result.fileCountLeft; } if (!file.isEmpty<RIGHT_SIDE>()) { - result.filesizeRightView += file.getFileSize<RIGHT_SIDE>(); - ++result.filesOnRightView; + result.bytesRight += file.getFileSize<RIGHT_SIDE>(); + ++result.fileCountRight; } }, [&](const SymlinkPair& symlink) { if (!symlink.isEmpty<LEFT_SIDE>()) - ++result.filesOnLeftView; + ++result.fileCountLeft; if (!symlink.isEmpty<RIGHT_SIDE>()) - ++result.filesOnRightView; + ++result.fileCountRight; }); } @@ -103,59 +103,53 @@ ptrdiff_t FileView::findRowFirstChild(const ContainerObject* hierObj) const FileView::StatusCmpResult FileView::updateCmpResult(bool showExcluded, //maps sortedRef to viewRef - bool leftOnlyFilesActive, - bool rightOnlyFilesActive, - bool leftNewerFilesActive, - bool rightNewerFilesActive, - bool differentFilesActive, - bool equalFilesActive, - bool conflictFilesActive) + bool showLeftOnly, + bool showRightOnly, + bool showLeftNewer, + bool showRightNewer, + bool showDifferent, + bool showEqual, + bool showConflict) { StatusCmpResult output; updateView([&](const FileSystemObject& fsObj) -> bool { - if (!fsObj.isActive()) + auto categorize = [&](bool showCategory, bool& existsCategory) { - output.existsExcluded = true; - if (!showExcluded) + if (!fsObj.isActive()) + { + output.existsExcluded = true; + if (!showExcluded) + return false; + } + existsCategory = true; + if (!showCategory) return false; - } + + addNumbers(fsObj, output); //calculate total number of bytes for each side + return true; + }; switch (fsObj.getCategory()) { case FILE_LEFT_SIDE_ONLY: - output.existsLeftOnly = true; - if (!leftOnlyFilesActive) return false; - break; + return categorize(showLeftOnly, output.existsLeftOnly); case FILE_RIGHT_SIDE_ONLY: - output.existsRightOnly = true; - if (!rightOnlyFilesActive) return false; - break; + return categorize(showRightOnly, output.existsRightOnly); case FILE_LEFT_NEWER: - output.existsLeftNewer = true; - if (!leftNewerFilesActive) return false; - break; + return categorize(showLeftNewer, output.existsLeftNewer); case FILE_RIGHT_NEWER: - output.existsRightNewer = true; - if (!rightNewerFilesActive) return false; - break; + return categorize(showRightNewer, output.existsRightNewer); case FILE_DIFFERENT_CONTENT: - output.existsDifferent = true; - if (!differentFilesActive) return false; - break; + return categorize(showDifferent, output.existsDifferent); case FILE_EQUAL: case FILE_DIFFERENT_METADATA: //= sub-category of equal - output.existsEqual = true; - if (!equalFilesActive) return false; - break; + return categorize(showEqual, output.existsEqual); case FILE_CONFLICT: - output.existsConflict = true; - if (!conflictFilesActive) return false; - break; + return categorize(showConflict, output.existsConflict); } - //calculate total number of bytes for each side - addNumbers(fsObj, output); + assert(false); return true; }); @@ -164,75 +158,64 @@ FileView::StatusCmpResult FileView::updateCmpResult(bool showExcluded, //maps so FileView::StatusSyncPreview FileView::updateSyncPreview(bool showExcluded, //maps sortedRef to viewRef - bool syncCreateLeftActive, - bool syncCreateRightActive, - bool syncDeleteLeftActive, - bool syncDeleteRightActive, - bool syncDirOverwLeftActive, - bool syncDirOverwRightActive, - bool syncDirNoneActive, - bool syncEqualActive, - bool conflictFilesActive) + bool showCreateLeft, + bool showCreateRight, + bool showDeleteLeft, + bool showDeleteRight, + bool showUpdateLeft, + bool showUpdateRight, + bool showDoNothing, + bool showEqual, + bool showConflict) { StatusSyncPreview output; updateView([&](const FileSystemObject& fsObj) { - if (!fsObj.isActive()) + auto categorize = [&](bool showCategory, bool& existsCategory) { - output.existsExcluded = true; - if (!showExcluded) + if (!fsObj.isActive()) + { + output.existsExcluded = true; + if (!showExcluded) + return false; + } + existsCategory = true; + if (!showCategory) return false; - } + + addNumbers(fsObj, output); //calculate total number of bytes for each side + return true; + }; switch (fsObj.getSyncOperation()) //evaluate comparison result and sync direction { case SO_CREATE_NEW_LEFT: - output.existsSyncCreateLeft = true; - if (!syncCreateLeftActive) return false; - break; + return categorize(showCreateLeft, output.existsSyncCreateLeft); case SO_CREATE_NEW_RIGHT: - output.existsSyncCreateRight = true; - if (!syncCreateRightActive) return false; - break; + return categorize(showCreateRight, output.existsSyncCreateRight); case SO_DELETE_LEFT: - output.existsSyncDeleteLeft = true; - if (!syncDeleteLeftActive) return false; - break; + return categorize(showDeleteLeft, output.existsSyncDeleteLeft); case SO_DELETE_RIGHT: - output.existsSyncDeleteRight = true; - if (!syncDeleteRightActive) return false; - break; + return categorize(showDeleteRight, output.existsSyncDeleteRight); case SO_OVERWRITE_RIGHT: case SO_COPY_METADATA_TO_RIGHT: //no extra button on screen case SO_MOVE_RIGHT_FROM: case SO_MOVE_RIGHT_TO: - output.existsSyncDirRight = true; - if (!syncDirOverwRightActive) return false; - break; + return categorize(showUpdateRight, output.existsSyncDirRight); case SO_OVERWRITE_LEFT: case SO_COPY_METADATA_TO_LEFT: //no extra button on screen case SO_MOVE_LEFT_TO: case SO_MOVE_LEFT_FROM: - output.existsSyncDirLeft = true; - if (!syncDirOverwLeftActive) return false; - break; + return categorize(showUpdateLeft, output.existsSyncDirLeft); case SO_DO_NOTHING: - output.existsSyncDirNone = true; - if (!syncDirNoneActive) return false; - break; + return categorize(showDoNothing, output.existsSyncDirNone); case SO_EQUAL: - output.existsEqual = true; - if (!syncEqualActive) return false; - break; + return categorize(showEqual, output.existsEqual); case SO_UNRESOLVED_CONFLICT: - output.existsConflict = true; - if (!conflictFilesActive) return false; - break; + return categorize(showConflict, output.existsConflict); } - - //calculate total number of bytes for each side - addNumbers(fsObj, output); + assert(false); return true; }); diff --git a/FreeFileSync/Source/ui/file_view.h b/FreeFileSync/Source/ui/file_view.h index 2bef757c..4dcb96a3 100755..100644 --- a/FreeFileSync/Source/ui/file_view.h +++ b/FreeFileSync/Source/ui/file_view.h @@ -43,24 +43,24 @@ public: bool existsRightNewer = false; bool existsDifferent = false; - unsigned int filesOnLeftView = 0; - unsigned int foldersOnLeftView = 0; - unsigned int filesOnRightView = 0; - unsigned int foldersOnRightView = 0; + unsigned int fileCountLeft = 0; + unsigned int folderCountLeft = 0; + uint64_t bytesLeft = 0; - uint64_t filesizeLeftView = 0; - uint64_t filesizeRightView = 0; + unsigned int fileCountRight = 0; + unsigned int folderCountRight = 0; + uint64_t bytesRight = 0; }; //comparison results view StatusCmpResult updateCmpResult(bool showExcluded, - bool leftOnlyFilesActive, - bool rightOnlyFilesActive, - bool leftNewerFilesActive, - bool rightNewerFilesActive, - bool differentFilesActive, - bool equalFilesActive, - bool conflictFilesActive); + bool showLeftOnly, + bool showRightOnly, + bool showLeftNewer, + bool showRightNewer, + bool showDifferent, + bool showEqual, + bool showConflict); struct StatusSyncPreview { @@ -76,26 +76,26 @@ public: bool existsSyncDirRight = false; bool existsSyncDirNone = false; - unsigned int filesOnLeftView = 0; - unsigned int foldersOnLeftView = 0; - unsigned int filesOnRightView = 0; - unsigned int foldersOnRightView = 0; + unsigned int fileCountLeft = 0; + unsigned int folderCountLeft = 0; + uint64_t bytesLeft = 0; - uint64_t filesizeLeftView = 0; - uint64_t filesizeRightView = 0; + unsigned int fileCountRight = 0; + unsigned int folderCountRight = 0; + uint64_t bytesRight = 0; }; //synchronization preview StatusSyncPreview updateSyncPreview(bool showExcluded, - bool syncCreateLeftActive, - bool syncCreateRightActive, - bool syncDeleteLeftActive, - bool syncDeleteRightActive, - bool syncDirOverwLeftActive, - bool syncDirOverwRightActive, - bool syncDirNoneActive, - bool syncEqualActive, - bool conflictFilesActive); + bool showCreateLeft, + bool showCreateRight, + bool showDeleteLeft, + bool showDeleteRight, + bool showUpdateLeft, + bool showUpdateRight, + bool showDoNothing, + bool showEqual, + bool showConflict); void setData(FolderComparison& newData); void removeInvalidRows(); //remove references to rows that have been deleted meanwhile: call after manual deletion and synchronization! diff --git a/FreeFileSync/Source/ui/folder_history_box.cpp b/FreeFileSync/Source/ui/folder_history_box.cpp index 932e6730..932e6730 100755..100644 --- a/FreeFileSync/Source/ui/folder_history_box.cpp +++ b/FreeFileSync/Source/ui/folder_history_box.cpp diff --git a/FreeFileSync/Source/ui/folder_history_box.h b/FreeFileSync/Source/ui/folder_history_box.h index e83346e1..e83346e1 100755..100644 --- a/FreeFileSync/Source/ui/folder_history_box.h +++ b/FreeFileSync/Source/ui/folder_history_box.h diff --git a/FreeFileSync/Source/ui/folder_pair.h b/FreeFileSync/Source/ui/folder_pair.h index 357de4d3..cb5794b6 100755..100644 --- a/FreeFileSync/Source/ui/folder_pair.h +++ b/FreeFileSync/Source/ui/folder_pair.h @@ -57,36 +57,40 @@ private: { using namespace zen; + const wxImage imgCmp = shrinkImage(getResourceImage(L"cfg_compare").ConvertToImage(), fastFromDIP(20)); + const wxImage imgSync = shrinkImage(getResourceImage(L"cfg_sync" ).ConvertToImage(), fastFromDIP(20)); + const wxImage imgFilter = shrinkImage(getResourceImage(L"cfg_filter" ).ConvertToImage(), fastFromDIP(20)); + if (localCmpCfg_) { - setImage(*basicPanel_.m_bpButtonLocalCompCfg, getResourceImage(L"cfg_compare_small")); + setImage(*basicPanel_.m_bpButtonLocalCompCfg, imgCmp); basicPanel_.m_bpButtonLocalCompCfg->SetToolTip(_("Local comparison settings") + L" (" + getVariantName(localCmpCfg_->compareVar) + L")"); } else { - setImage(*basicPanel_.m_bpButtonLocalCompCfg, greyScale(getResourceImage(L"cfg_compare_small"))); + setImage(*basicPanel_.m_bpButtonLocalCompCfg, greyScale(imgCmp)); basicPanel_.m_bpButtonLocalCompCfg->SetToolTip(_("Local comparison settings")); } if (localSyncCfg_) { - setImage(*basicPanel_.m_bpButtonLocalSyncCfg, getResourceImage(L"cfg_sync_small")); + setImage(*basicPanel_.m_bpButtonLocalSyncCfg, imgSync); basicPanel_.m_bpButtonLocalSyncCfg->SetToolTip(_("Local synchronization settings") + L" (" + getVariantName(localSyncCfg_->directionCfg.var) + L")"); } else { - setImage(*basicPanel_.m_bpButtonLocalSyncCfg, greyScale(getResourceImage(L"cfg_sync_small"))); + setImage(*basicPanel_.m_bpButtonLocalSyncCfg, greyScale(imgSync)); basicPanel_.m_bpButtonLocalSyncCfg->SetToolTip(_("Local synchronization settings")); } if (!isNullFilter(localFilter_)) { - setImage(*basicPanel_.m_bpButtonLocalFilter, getResourceImage(L"cfg_filter_small")); + setImage(*basicPanel_.m_bpButtonLocalFilter, imgFilter); basicPanel_.m_bpButtonLocalFilter->SetToolTip(_("Local filter") + L" (" + _("Active") + L")"); } else { - setImage(*basicPanel_.m_bpButtonLocalFilter, greyScale(getResourceImage(L"cfg_filter_small"))); + setImage(*basicPanel_.m_bpButtonLocalFilter, greyScale(imgFilter)); basicPanel_.m_bpButtonLocalFilter->SetToolTip(_("Local filter") + L" (" + _("None") + L")"); } } diff --git a/FreeFileSync/Source/ui/folder_selector.cpp b/FreeFileSync/Source/ui/folder_selector.cpp index 7f44e50e..86ae2e16 100755..100644 --- a/FreeFileSync/Source/ui/folder_selector.cpp +++ b/FreeFileSync/Source/ui/folder_selector.cpp @@ -17,7 +17,7 @@ #include "../base/icon_buffer.h" - using AFS = fff::AbstractFileSystem; + #include "small_dlgs.h" //includes structures.h, which defines "AFS" using namespace zen; using namespace fff; @@ -57,7 +57,8 @@ const wxEventType fff::EVENT_ON_FOLDER_SELECTED = wxNewEventType(); const wxEventType fff::EVENT_ON_FOLDER_MANUAL_EDIT = wxNewEventType(); -FolderSelector::FolderSelector(wxWindow& dropWindow, +FolderSelector::FolderSelector(wxWindow* parent, + wxWindow& dropWindow, wxButton& selectFolderButton, wxButton& selectAltFolderButton, FolderHistoryBox& folderComboBox, @@ -69,6 +70,7 @@ FolderSelector::FolderSelector(wxWindow& dropWindow, droppedPathsFilter_ (droppedPathsFilter), getDeviceParallelOps_(getDeviceParallelOps), setDeviceParallelOps_(setDeviceParallelOps), + parent_(parent), dropWindow_(dropWindow), dropWindow2_(dropWindow2), selectFolderButton_(selectFolderButton), @@ -87,7 +89,7 @@ FolderSelector::FolderSelector(wxWindow& dropWindow, setupDragDrop(dropWindow_); if (dropWindow2_) setupDragDrop(*dropWindow2_); - selectAltFolderButton_.Hide(); + selectAltFolderButton_.SetBitmapLabel(getResourceImage(L"cloud_small")); //keep dirPicker and dirpath synchronous folderComboBox_ .Connect(wxEVT_MOUSEWHEEL, wxMouseEventHandler (FolderSelector::onMouseWheel ), nullptr, this); @@ -205,8 +207,7 @@ void FolderSelector::onSelectFolder(wxCommandEvent& event) } } - //wxDirDialog internally uses lame-looking SHBrowseForFolder(); we better use IFileDialog() instead! (remembers size and position!) - wxDirDialog dirPicker(&selectFolderButton_, _("Select a folder"), utfTo<wxString>(defaultFolderPath)); //put modal wxWidgets dialogs on stack: creating on freestore leads to memleak! + wxDirDialog dirPicker(parent_, _("Select a folder"), utfTo<wxString>(defaultFolderPath)); //put modal wxWidgets dialogs on stack: creating on freestore leads to memleak! //-> following doesn't seem to do anything at all! still "Show hidden" is available as a context menu option: //::gtk_file_chooser_set_show_hidden(GTK_FILE_CHOOSER(dirPicker.m_widget), true /*show_hidden*/); @@ -225,6 +226,24 @@ void FolderSelector::onSelectFolder(wxCommandEvent& event) void FolderSelector::onSelectAltFolder(wxCommandEvent& event) { + Zstring folderPathPhrase = getPath(); + size_t parallelOps = getDeviceParallelOps_ ? getDeviceParallelOps_(folderPathPhrase) : 1; + + std::optional<std::wstring> parallelOpsDisabledReason; + + parallelOpsDisabledReason = _("Requires FreeFileSync Donation Edition"); + + if (showCloudSetupDialog(parent_, folderPathPhrase, parallelOps, get(parallelOpsDisabledReason)) != ReturnSmallDlg::BUTTON_OKAY) + return; + + setFolderPathPhrase(folderPathPhrase, &folderComboBox_, folderComboBox_, staticText_); + + if (setDeviceParallelOps_) + setDeviceParallelOps_(folderPathPhrase, parallelOps); + + //notify action invoked by user + wxCommandEvent dummy(EVENT_ON_FOLDER_SELECTED); + ProcessEvent(dummy); } diff --git a/FreeFileSync/Source/ui/folder_selector.h b/FreeFileSync/Source/ui/folder_selector.h index 01ca8e2c..3039cd40 100755..100644 --- a/FreeFileSync/Source/ui/folder_selector.h +++ b/FreeFileSync/Source/ui/folder_selector.h @@ -33,7 +33,8 @@ extern const wxEventType EVENT_ON_FOLDER_MANUAL_EDIT; //manual type-in class FolderSelector: public wxEvtHandler { public: - FolderSelector(wxWindow& dropWindow, + FolderSelector(wxWindow* parent, + wxWindow& dropWindow, wxButton& selectFolderButton, wxButton& selectAltFolderButton, FolderHistoryBox& folderComboBox, @@ -63,6 +64,7 @@ private: const std::function<size_t(const Zstring& folderPathPhrase)> getDeviceParallelOps_; const std::function<void (const Zstring& folderPathPhrase, size_t parallelOps)> setDeviceParallelOps_; + wxWindow* parent_; wxWindow& dropWindow_; wxWindow* dropWindow2_ = nullptr; wxButton& selectFolderButton_; diff --git a/FreeFileSync/Source/ui/gui_generated.cpp b/FreeFileSync/Source/ui/gui_generated.cpp index 22919575..5e2cb523 100755..100644 --- a/FreeFileSync/Source/ui/gui_generated.cpp +++ b/FreeFileSync/Source/ui/gui_generated.cpp @@ -853,21 +853,16 @@ MainDialogGenerated::MainDialogGenerated( wxWindow* parent, wxWindowID id, const m_bpButtonViewTypeSyncAction = new zen::ToggleButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxSize( -1,-1 ), wxBU_AUTODRAW ); bSizerViewFilter->Add( m_bpButtonViewTypeSyncAction, 0, wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL|wxRIGHT, 5 ); - m_bpButtonShowExcluded = new zen::ToggleButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxSize( -1,-1 ), wxBU_AUTODRAW ); - bSizerViewFilter->Add( m_bpButtonShowExcluded, 0, wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL, 5 ); - - - bSizerViewFilter->Add( 0, 0, 1, wxEXPAND, 5 ); - m_bpButtonViewFilterSave = new wxBitmapButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxDefaultSize, wxBU_AUTODRAW ); - m_bpButtonViewFilterSave->SetToolTip( _("Save as default") ); - - bSizerViewFilter->Add( m_bpButtonViewFilterSave, 0, wxALIGN_CENTER_VERTICAL|wxTOP|wxBOTTOM|wxLEFT, 5 ); + bSizerViewFilter->Add( 0, 0, 3, wxEXPAND, 5 ); m_staticTextSelectView = new wxStaticText( m_panelViewFilter, wxID_ANY, _("Select view:"), wxDefaultPosition, wxDefaultSize, 0 ); m_staticTextSelectView->Wrap( -1 ); bSizerViewFilter->Add( m_staticTextSelectView, 0, wxALL|wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL, 5 ); + m_bpButtonShowExcluded = new zen::ToggleButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxSize( -1,-1 ), wxBU_AUTODRAW ); + bSizerViewFilter->Add( m_bpButtonShowExcluded, 0, wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL|wxRIGHT, 5 ); + m_bpButtonShowDeleteLeft = new zen::ToggleButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxSize( -1,-1 ), wxBU_AUTODRAW ); bSizerViewFilter->Add( m_bpButtonShowDeleteLeft, 0, wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL, 5 ); @@ -910,8 +905,13 @@ MainDialogGenerated::MainDialogGenerated( wxWindow* parent, wxWindowID id, const m_bpButtonShowConflict = new zen::ToggleButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxSize( -1,-1 ), wxBU_AUTODRAW ); bSizerViewFilter->Add( m_bpButtonShowConflict, 0, wxALIGN_CENTER_VERTICAL|wxALIGN_CENTER_HORIZONTAL, 5 ); + m_bpButtonViewFilterSave = new wxBitmapButton( m_panelViewFilter, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxDefaultSize, wxBU_AUTODRAW ); + m_bpButtonViewFilterSave->SetToolTip( _("Save as default") ); - bSizerViewFilter->Add( 0, 0, 1, wxEXPAND, 5 ); + bSizerViewFilter->Add( m_bpButtonViewFilterSave, 0, wxALIGN_BOTTOM|wxTOP|wxBOTTOM|wxRIGHT, 5 ); + + + bSizerViewFilter->Add( 0, 0, 3, wxEXPAND, 5 ); m_staticText96 = new wxStaticText( m_panelViewFilter, wxID_ANY, _("Statistics:"), wxDefaultPosition, wxDefaultSize, 0 ); m_staticText96->Wrap( -1 ); @@ -1174,7 +1174,6 @@ MainDialogGenerated::MainDialogGenerated( wxWindow* parent, wxWindowID id, const m_bpButtonShowLog->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnShowLog ), NULL, this ); m_bpButtonViewTypeSyncAction->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewType ), NULL, this ); m_bpButtonShowExcluded->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); - m_bpButtonViewFilterSave->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnViewFilterSave ), NULL, this ); m_bpButtonShowDeleteLeft->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); m_bpButtonShowUpdateLeft->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); m_bpButtonShowCreateLeft->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); @@ -1189,6 +1188,7 @@ MainDialogGenerated::MainDialogGenerated( wxWindow* parent, wxWindowID id, const m_bpButtonShowUpdateRight->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); m_bpButtonShowDeleteRight->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); m_bpButtonShowConflict->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnToggleViewButton ), NULL, this ); + m_bpButtonViewFilterSave->Connect( wxEVT_COMMAND_BUTTON_CLICKED, wxCommandEventHandler( MainDialogGenerated::OnViewFilterSave ), NULL, this ); } MainDialogGenerated::~MainDialogGenerated() @@ -2507,7 +2507,7 @@ CloudSetupDlgGenerated::CloudSetupDlgGenerated( wxWindow* parent, wxWindowID id, bSizer285->Add( m_staticText166, 0, wxTOP|wxRIGHT|wxLEFT, 5 ); m_listBoxGdriveUsers = new wxListBox( m_panel41, wxID_ANY, wxDefaultPosition, wxDefaultSize, 0, NULL, wxLB_NEEDED_SB|wxLB_SINGLE|wxLB_SORT ); - bSizer285->Add( m_listBoxGdriveUsers, 1, wxTOP|wxBOTTOM|wxLEFT, 5 ); + bSizer285->Add( m_listBoxGdriveUsers, 1, wxTOP|wxBOTTOM|wxLEFT|wxEXPAND, 5 ); bSizer284->Add( bSizer285, 0, wxEXPAND|wxTOP|wxBOTTOM|wxLEFT, 5 ); @@ -2946,28 +2946,16 @@ AbstractFolderPickerGenerated::AbstractFolderPickerGenerated( wxWindow* parent, wxBoxSizer* bSizer134; bSizer134 = new wxBoxSizer( wxVERTICAL ); - wxBoxSizer* bSizer72; - bSizer72 = new wxBoxSizer( wxHORIZONTAL ); - - m_bitmapServer = new wxStaticBitmap( this, wxID_ANY, wxNullBitmap, wxDefaultPosition, wxDefaultSize, 0 ); - bSizer72->Add( m_bitmapServer, 0, wxALIGN_CENTER_VERTICAL|wxTOP|wxBOTTOM|wxLEFT, 10 ); - - m_staticTextHeader = new wxStaticText( this, wxID_ANY, _("Select a directory on the server:"), wxDefaultPosition, wxDefaultSize, 0 ); - m_staticTextHeader->Wrap( -1 ); - bSizer72->Add( m_staticTextHeader, 0, wxALIGN_CENTER_HORIZONTAL|wxALIGN_CENTER_VERTICAL|wxALL, 10 ); - - - bSizer134->Add( bSizer72, 0, 0, 5 ); - - m_staticline371 = new wxStaticLine( this, wxID_ANY, wxDefaultPosition, wxSize( -1,-1 ), wxLI_HORIZONTAL ); - bSizer134->Add( m_staticline371, 0, wxEXPAND, 5 ); - m_panel41 = new wxPanel( this, wxID_ANY, wxDefaultPosition, wxDefaultSize, wxTAB_TRAVERSAL ); m_panel41->SetBackgroundColour( wxSystemSettings::GetColour( wxSYS_COLOUR_WINDOW ) ); wxBoxSizer* bSizer185; bSizer185 = new wxBoxSizer( wxVERTICAL ); + m_staticTextStatus = new wxStaticText( m_panel41, wxID_ANY, _("dummy"), wxDefaultPosition, wxDefaultSize, 0 ); + m_staticTextStatus->Wrap( -1 ); + bSizer185->Add( m_staticTextStatus, 0, wxALL, 5 ); + m_treeCtrlFileSystem = new wxTreeCtrl( m_panel41, wxID_ANY, wxDefaultPosition, wxSize( -1,-1 ), wxTR_FULL_ROW_HIGHLIGHT|wxTR_HAS_BUTTONS|wxTR_LINES_AT_ROOT|wxTR_NO_LINES|wxNO_BORDER ); bSizer185->Add( m_treeCtrlFileSystem, 1, wxEXPAND, 5 ); diff --git a/FreeFileSync/Source/ui/gui_generated.h b/FreeFileSync/Source/ui/gui_generated.h index 7523ea04..635172c1 100755..100644 --- a/FreeFileSync/Source/ui/gui_generated.h +++ b/FreeFileSync/Source/ui/gui_generated.h @@ -172,9 +172,8 @@ class MainDialogGenerated : public wxFrame wxBitmapButton* m_bpButtonShowLog; wxStaticText* m_staticTextViewType; zen::ToggleButton* m_bpButtonViewTypeSyncAction; - zen::ToggleButton* m_bpButtonShowExcluded; - wxBitmapButton* m_bpButtonViewFilterSave; wxStaticText* m_staticTextSelectView; + zen::ToggleButton* m_bpButtonShowExcluded; zen::ToggleButton* m_bpButtonShowDeleteLeft; zen::ToggleButton* m_bpButtonShowUpdateLeft; zen::ToggleButton* m_bpButtonShowCreateLeft; @@ -189,6 +188,7 @@ class MainDialogGenerated : public wxFrame zen::ToggleButton* m_bpButtonShowUpdateRight; zen::ToggleButton* m_bpButtonShowDeleteRight; zen::ToggleButton* m_bpButtonShowConflict; + wxBitmapButton* m_bpButtonViewFilterSave; wxStaticText* m_staticText96; wxPanel* m_panelStatistics; wxBoxSizer* bSizer1801; @@ -657,10 +657,8 @@ class AbstractFolderPickerGenerated : public wxDialog private: protected: - wxStaticBitmap* m_bitmapServer; - wxStaticText* m_staticTextHeader; - wxStaticLine* m_staticline371; wxPanel* m_panel41; + wxStaticText* m_staticTextStatus; wxTreeCtrl* m_treeCtrlFileSystem; wxStaticLine* m_staticline12; wxBoxSizer* bSizerStdButtons; @@ -676,7 +674,7 @@ class AbstractFolderPickerGenerated : public wxDialog public: - AbstractFolderPickerGenerated( wxWindow* parent, wxWindowID id = wxID_ANY, const wxString& title = wxEmptyString, const wxPoint& pos = wxDefaultPosition, const wxSize& size = wxSize( -1,-1 ), long style = wxDEFAULT_DIALOG_STYLE|wxMAXIMIZE_BOX|wxRESIZE_BORDER ); + AbstractFolderPickerGenerated( wxWindow* parent, wxWindowID id = wxID_ANY, const wxString& title = _("Select a folder"), const wxPoint& pos = wxDefaultPosition, const wxSize& size = wxSize( -1,-1 ), long style = wxDEFAULT_DIALOG_STYLE|wxMAXIMIZE_BOX|wxRESIZE_BORDER ); ~AbstractFolderPickerGenerated(); }; diff --git a/FreeFileSync/Source/ui/gui_status_handler.cpp b/FreeFileSync/Source/ui/gui_status_handler.cpp index 52d6ff69..52d6ff69 100755..100644 --- a/FreeFileSync/Source/ui/gui_status_handler.cpp +++ b/FreeFileSync/Source/ui/gui_status_handler.cpp diff --git a/FreeFileSync/Source/ui/gui_status_handler.h b/FreeFileSync/Source/ui/gui_status_handler.h index 0bd6dac4..0bd6dac4 100755..100644 --- a/FreeFileSync/Source/ui/gui_status_handler.h +++ b/FreeFileSync/Source/ui/gui_status_handler.h diff --git a/FreeFileSync/Source/ui/log_panel.cpp b/FreeFileSync/Source/ui/log_panel.cpp index fca60a36..fca60a36 100755..100644 --- a/FreeFileSync/Source/ui/log_panel.cpp +++ b/FreeFileSync/Source/ui/log_panel.cpp diff --git a/FreeFileSync/Source/ui/log_panel.h b/FreeFileSync/Source/ui/log_panel.h index 47f9a84a..47f9a84a 100755..100644 --- a/FreeFileSync/Source/ui/log_panel.h +++ b/FreeFileSync/Source/ui/log_panel.h diff --git a/FreeFileSync/Source/ui/main_dlg.cpp b/FreeFileSync/Source/ui/main_dlg.cpp index 55e20fbd..d998c55f 100755..100644 --- a/FreeFileSync/Source/ui/main_dlg.cpp +++ b/FreeFileSync/Source/ui/main_dlg.cpp @@ -122,8 +122,8 @@ public: wxWindow* dropWindow2R) : FolderPairPanelBasic<GuiPanel>(basicPanel), //pass FolderPairPanelGenerated part... mainDlg_(mainDialog), - folderSelectorLeft_ (dropWindow1L, selectFolderButtonL, selectSftpButtonL, dirpathL, staticTextL, dropWindow2L, droppedPathsFilter_, getDeviceParallelOps_, setDeviceParallelOps_), - folderSelectorRight_(dropWindow1R, selectFolderButtonR, selectSftpButtonR, dirpathR, staticTextR, dropWindow2R, droppedPathsFilter_, getDeviceParallelOps_, setDeviceParallelOps_) + folderSelectorLeft_ (&mainDialog, dropWindow1L, selectFolderButtonL, selectSftpButtonL, dirpathL, staticTextL, dropWindow2L, droppedPathsFilter_, getDeviceParallelOps_, setDeviceParallelOps_), + folderSelectorRight_(&mainDialog, dropWindow1R, selectFolderButtonR, selectSftpButtonR, dirpathR, staticTextR, dropWindow2R, droppedPathsFilter_, getDeviceParallelOps_, setDeviceParallelOps_) { folderSelectorLeft_ .setSiblingSelector(&folderSelectorRight_); folderSelectorRight_.setSiblingSelector(&folderSelectorLeft_); @@ -412,6 +412,7 @@ MainDialog::MainDialog(const Zstring& globalConfigFilePath, m_bpButtonViewFilterSave->SetBitmapLabel(getResourceImage(L"file_save_sicon")); + m_bpButtonFilter ->SetMinSize(wxSize(getResourceImage(L"cfg_filter").GetWidth() + fastFromDIP(27), -1)); //make the filter button wider m_textCtrlSearchTxt->SetMinSize(wxSize(fastFromDIP(220), -1)); initViewFilterButtons(); @@ -653,7 +654,10 @@ MainDialog::MainDialog(const Zstring& globalConfigFilePath, wxMenuItem* newItem = new wxMenuItem(menu, wxID_ANY, _("&Show details")); this->Connect(newItem->GetId(), wxEVT_COMMAND_MENU_SELECTED, wxCommandEventHandler(MainDialog::OnMenuUpdateAvailable)); menu->Append(newItem); //pass ownership - m_menubar->Append(menu, L"\u2605 " + replaceCpy(_("FreeFileSync %x is available!"), L"%x", utfTo<std::wstring>(globalSettings.gui.lastOnlineVersion)) + L" \u2605"); //"BLACK STAR" + + const std::wstring blackStar = utfTo<std::wstring>("\xF0\x9F\x9F\x8A"); //"HEAVY FIVE POINTED BLACK STAR" + m_menubar->Append(menu, blackStar + L" " + replaceCpy(_("FreeFileSync %x is available!"), L"%x", utfTo<std::wstring>(globalSettings.gui.lastOnlineVersion)) + L" " + blackStar); + } //notify about (logical) application main window => program won't quit, but stay on this dialog @@ -902,7 +906,7 @@ void MainDialog::setGlobalCfgOnInit(const XmlGlobalSettings& globalSettings) //old comment: "wxGTK's wxWindow::SetSize seems unreliable and behaves like a wxWindow::SetClientSize // => use wxWindow::SetClientSize instead (for the record: no such issue on Windows/OS X) - //2018-10-15: Weird new problem on Centos/Ubuntu: SetClientSize() + SetPosition() fail to set correct dialog *position*, but SetSize() + SetPosition() do! + //2018-10-15: Weird new problem on CentOS/Ubuntu: SetClientSize() + SetPosition() fail to set correct dialog *position*, but SetSize() + SetPosition() do! // => old issues with SetSize() seem to be gone... => revert to SetSize() if (newPos) SetSize(wxRect(*newPos, newSize)); @@ -914,11 +918,6 @@ void MainDialog::setGlobalCfgOnInit(const XmlGlobalSettings& globalSettings) if (globalSettings.gui.mainDlg.isMaximized) //no real need to support both maximize and full screen functions { - //#ifdef ZEN_MAC - // if (fullScreenApiSupported) -> starting in full screen seems to annoy users - // ShowFullScreen(true); //once EnableFullScreenView() is set, this internally uses the new full screen API - // else - //#endif Maximize(true); } @@ -1434,7 +1433,7 @@ void MainDialog::openExternalApplication(const Zstring& commandLinePhrase, bool //regular command evaluation: const size_t invokeCount = selectionLeft.size() + selectionRight.size(); if (invokeCount > EXT_APP_MASS_INVOKE_THRESHOLD) - if (globalCfg_.confirmDlgs.confirmExternalCommandMassInvoke) + if (globalCfg_.confirmDlgs.confirmCommandMassInvoke) { bool dontAskAgain = false; switch (showConfirmationDialog(this, DialogInfoType::WARNING, PopupDialogCfg(). @@ -1446,7 +1445,7 @@ void MainDialog::openExternalApplication(const Zstring& commandLinePhrase, bool _("&Execute"))) { case ConfirmationButton::ACCEPT: - globalCfg_.confirmDlgs.confirmExternalCommandMassInvoke = !dontAskAgain; + globalCfg_.confirmDlgs.confirmCommandMassInvoke = !dontAskAgain; break; case ConfirmationButton::CANCEL: return; @@ -1510,12 +1509,12 @@ void MainDialog::openExternalApplication(const Zstring& commandLinePhrase, bool } -void MainDialog::setStatusBarFileStatistics(size_t filesOnLeftView, - size_t foldersOnLeftView, - size_t filesOnRightView, - size_t foldersOnRightView, - uint64_t filesizeLeftView, - uint64_t filesizeRightView) +void MainDialog::setStatusBarFileStats(size_t fileCountLeft, + size_t folderCountLeft, + uint64_t bytesLeft, + size_t fileCountRight, + size_t folderCountRight, + uint64_t bytesRight) { //select state @@ -1523,19 +1522,19 @@ void MainDialog::setStatusBarFileStatistics(size_t filesOnLeftView, m_staticTextFullStatus->Hide(); //update status information - bSizerStatusLeftDirectories->Show(foldersOnLeftView > 0); - bSizerStatusLeftFiles ->Show(filesOnLeftView > 0); + bSizerStatusLeftDirectories->Show(folderCountLeft > 0); + bSizerStatusLeftFiles ->Show(fileCountLeft > 0); - setText(*m_staticTextStatusLeftDirs, _P("1 directory", "%x directories", foldersOnLeftView)); - setText(*m_staticTextStatusLeftFiles, _P("1 file", "%x files", filesOnLeftView)); - setText(*m_staticTextStatusLeftBytes, L"(" + formatFilesizeShort(filesizeLeftView) + L")"); + setText(*m_staticTextStatusLeftDirs, _P("1 directory", "%x directories", folderCountLeft)); + setText(*m_staticTextStatusLeftFiles, _P("1 file", "%x files", fileCountLeft)); + setText(*m_staticTextStatusLeftBytes, L"(" + formatFilesizeShort(bytesLeft) + L")"); //------------------------------------------------------------------------------ - bSizerStatusRightDirectories->Show(foldersOnRightView > 0); - bSizerStatusRightFiles ->Show(filesOnRightView > 0); + bSizerStatusRightDirectories->Show(folderCountRight > 0); + bSizerStatusRightFiles ->Show(fileCountRight > 0); - setText(*m_staticTextStatusRightDirs, _P("1 directory", "%x directories", foldersOnRightView)); - setText(*m_staticTextStatusRightFiles, _P("1 file", "%x files", filesOnRightView)); - setText(*m_staticTextStatusRightBytes, L"(" + formatFilesizeShort(filesizeRightView) + L")"); + setText(*m_staticTextStatusRightDirs, _P("1 directory", "%x directories", folderCountRight)); + setText(*m_staticTextStatusRightFiles, _P("1 file", "%x files", fileCountRight)); + setText(*m_staticTextStatusRightBytes, L"(" + formatFilesizeShort(bytesRight) + L")"); //------------------------------------------------------------------------------ wxString statusCenterNew; if (filegrid::getDataView(*m_gridMainC).rowsTotal() > 0) @@ -2823,14 +2822,14 @@ void MainDialog::OnSaveAsBatchJob(wxCommandEvent& event) } -bool MainDialog::trySaveConfig(const Zstring* guiFilename) //return true if saved successfully +bool MainDialog::trySaveConfig(const Zstring* guiCfgPath) //return true if saved successfully { - Zstring targetFilename; + Zstring cfgFilePath; - if (guiFilename) + if (guiCfgPath) { - targetFilename = *guiFilename; - assert(endsWith(targetFilename, Zstr(".ffs_gui"))); + cfgFilePath = *guiCfgPath; + assert(endsWith(cfgFilePath, Zstr(".ffs_gui"))); } else { @@ -2848,15 +2847,15 @@ bool MainDialog::trySaveConfig(const Zstring* guiFilename) //return true if save wxFD_SAVE | wxFD_OVERWRITE_PROMPT); if (filePicker.ShowModal() != wxID_OK) return false; - targetFilename = utfTo<Zstring>(filePicker.GetPath()); + cfgFilePath = utfTo<Zstring>(filePicker.GetPath()); } const XmlGuiConfig guiCfg = getConfig(); try { - writeConfig(guiCfg, targetFilename); //throw FileError - setLastUsedConfig(targetFilename, guiCfg); + writeConfig(guiCfg, cfgFilePath); //throw FileError + setLastUsedConfig({ cfgFilePath }, guiCfg); flashStatusInformation(_("Configuration saved")); return true; @@ -2869,7 +2868,7 @@ bool MainDialog::trySaveConfig(const Zstring* guiFilename) //return true if save } -bool MainDialog::trySaveBatchConfig(const Zstring* batchFileToUpdate) +bool MainDialog::trySaveBatchConfig(const Zstring* batchCfgPath) { //essentially behave like trySaveConfig(): the collateral damage of not saving GUI-only settings "m_bpButtonViewTypeSyncAction" is negligible @@ -2880,8 +2879,8 @@ bool MainDialog::trySaveBatchConfig(const Zstring* batchFileToUpdate) try { Zstring referenceBatchFile; - if (batchFileToUpdate) - referenceBatchFile = *batchFileToUpdate; + if (batchCfgPath) + referenceBatchFile = *batchCfgPath; else if (!activeCfgFilePath.empty()) if (getXmlType(activeCfgFilePath) == XML_TYPE_BATCH) //throw FileError referenceBatchFile = activeCfgFilePath; @@ -2902,11 +2901,11 @@ bool MainDialog::trySaveBatchConfig(const Zstring* batchFileToUpdate) return false; } - Zstring targetFilename; - if (batchFileToUpdate) + Zstring cfgFilePath; + if (batchCfgPath) { - targetFilename = *batchFileToUpdate; - assert(endsWith(targetFilename, Zstr(".ffs_batch"))); + cfgFilePath = *batchCfgPath; + assert(endsWith(cfgFilePath, Zstr(".ffs_batch"))); } else { @@ -2931,7 +2930,7 @@ bool MainDialog::trySaveBatchConfig(const Zstring* batchFileToUpdate) wxFD_SAVE | wxFD_OVERWRITE_PROMPT); if (filePicker.ShowModal() != wxID_OK) return false; - targetFilename = utfTo<Zstring>(filePicker.GetPath()); + cfgFilePath = utfTo<Zstring>(filePicker.GetPath()); } const XmlGuiConfig guiCfg = getConfig(); @@ -2939,8 +2938,8 @@ bool MainDialog::trySaveBatchConfig(const Zstring* batchFileToUpdate) try { - writeConfig(batchCfg, targetFilename); //throw FileError - setLastUsedConfig(targetFilename, guiCfg); //[!] behave as if we had saved guiCfg + writeConfig(batchCfg, cfgFilePath); //throw FileError + setLastUsedConfig({ cfgFilePath }, guiCfg); //[!] behave as if we had saved guiCfg flashStatusInformation(_("Configuration saved")); return true; @@ -3005,7 +3004,7 @@ bool MainDialog::saveOldConfig() //return false on user abort } //discard current reference file(s), this ensures next app start will load <last session> instead of the original non-modified config selection - setLastUsedConfig(std::vector<Zstring>(), lastSavedCfg_); + setLastUsedConfig({} /*cfgFilePaths*/, lastSavedCfg_); //this seems to make theoretical sense also: the job of this function is to make sure current (volatile) config and reference file name are in sync // => if user does not save cfg, it is not attached to a physical file names anymore! } @@ -3127,15 +3126,18 @@ void MainDialog::deleteSelectedCfgHistoryItems() cfggrid::getDataView(*m_gridCfgHistory).removeItems(filePaths); m_gridCfgHistory->Refresh(); //grid size changed => clears selection! - //set active selection on next element to allow "batch-deletion" by holding down DEL key + //set active selection on next item to allow "batch-deletion" by holding down DEL key + //user expects that selected config is also loaded: https://freefilesync.org/forum/viewtopic.php?t=5723 + std::vector<Zstring> nextCfgPaths; if (m_gridCfgHistory->getRowCount() > 0) { - size_t nextRow = selectedRows.front(); - if (nextRow >= m_gridCfgHistory->getRowCount()) - nextRow = m_gridCfgHistory->getRowCount() - 1; - - m_gridCfgHistory->selectRow(nextRow, GridEventPolicy::DENY); + const size_t nextRow = std::min(selectedRows.front(), m_gridCfgHistory->getRowCount() - 1); + if (const ConfigView::Details* cfg = cfggrid::getDataView(*m_gridCfgHistory).getItem(nextRow)) + nextCfgPaths.push_back(cfg->cfgItem.cfgFilePath ); } + + if (!loadConfiguration(nextCfgPaths)) + setConfig(currentCfg_, {}); //error/cancel => clear "activeConfigFiles_" } } @@ -3626,10 +3628,10 @@ void MainDialog::OnViewFilterSave(wxCommandEvent& event) setButtonDefault(m_bpButtonShowCreateLeft, def.createLeft); setButtonDefault(m_bpButtonShowCreateRight, def.createRight); - setButtonDefault(m_bpButtonShowUpdateLeft, def.updateLeft); - setButtonDefault(m_bpButtonShowUpdateRight, def.updateRight); setButtonDefault(m_bpButtonShowDeleteLeft, def.deleteLeft); setButtonDefault(m_bpButtonShowDeleteRight, def.deleteRight); + setButtonDefault(m_bpButtonShowUpdateLeft, def.updateLeft); + setButtonDefault(m_bpButtonShowUpdateRight, def.updateRight); setButtonDefault(m_bpButtonShowDoNothing, def.doNothing); }; @@ -3682,7 +3684,6 @@ void MainDialog::OnCompare(wxCommandEvent& event) const auto& guiCfg = getConfig(); const std::chrono::system_clock::time_point startTime = std::chrono::system_clock::now(); - const std::map<AfsDevice, size_t>& deviceParallelOps = guiCfg.mainCfg.deviceParallelOps; //handle status display and error messages StatusHandlerTemporaryPanel statusHandler(*this, startTime, @@ -3702,7 +3703,6 @@ void MainDialog::OnCompare(wxCommandEvent& event) globalCfg_.createLockFile, dirLocks, extractCompareCfg(guiCfg.mainCfg), - deviceParallelOps, statusHandler); //throw AbortProcess } catch (AbortProcess&) {} @@ -3772,8 +3772,7 @@ void MainDialog::updateGui() m_menuItemExportList->Enable(!folderCmp_.empty()); //a CSV without even folder names confuses users: https://freefilesync.org/forum/viewtopic.php?t=4787 - warn_static("still needed???") - //auiMgr_.Update(); //fix small display distortion, if view filter panel is empty + //auiMgr_.Update(); -> doesn't seem to be needed } @@ -3879,7 +3878,6 @@ void MainDialog::OnStartSync(wxCommandEvent& event) globalCfg_.confirmDlgs.confirmSyncStart = !dontShowAgain; } - const std::map<AfsDevice, size_t>& deviceParallelOps = guiCfg.mainCfg.deviceParallelOps; std::set<AbstractPath> logFilePathsToKeep; for (const ConfigFileItem& item : cfggrid::getDataView(*m_gridCfgHistory).get()) @@ -3943,7 +3941,6 @@ void MainDialog::OnStartSync(wxCommandEvent& event) globalCfg_.runWithBackgroundPriority, extractSyncCfg(guiCfg.mainCfg), folderCmp_, - deviceParallelOps, globalCfg_.warnDlgs, statusHandler); //throw AbortProcess } @@ -4247,12 +4244,13 @@ void MainDialog::OnSwapSides(wxCommandEvent& event) void MainDialog::updateGridViewData() { - size_t filesOnLeftView = 0; - size_t foldersOnLeftView = 0; - size_t filesOnRightView = 0; - size_t foldersOnRightView = 0; - uint64_t filesizeLeftView = 0; - uint64_t filesizeRightView = 0; + size_t fileCountLeft = 0; + size_t folderCountLeft = 0; + uint64_t bytesLeft = 0; + + size_t fileCountRight = 0; + size_t folderCountRight = 0; + uint64_t bytesRight = 0; auto updateVisibility = [](ToggleButton* btn, bool shown) { @@ -4272,12 +4270,13 @@ void MainDialog::updateGridViewData() m_bpButtonShowDoNothing ->isActive(), m_bpButtonShowEqual ->isActive(), m_bpButtonShowConflict ->isActive()); - filesOnLeftView = result.filesOnLeftView; - foldersOnLeftView = result.foldersOnLeftView; - filesOnRightView = result.filesOnRightView; - foldersOnRightView = result.foldersOnRightView; - filesizeLeftView = result.filesizeLeftView; - filesizeRightView = result.filesizeRightView; + fileCountLeft = result.fileCountLeft; + folderCountLeft = result.folderCountLeft; + bytesLeft = result.bytesLeft; + + fileCountRight = result.fileCountRight; + folderCountRight = result.folderCountRight; + bytesRight = result.bytesRight; //sync preview buttons updateVisibility(m_bpButtonShowExcluded, result.existsExcluded); @@ -4308,12 +4307,13 @@ void MainDialog::updateGridViewData() m_bpButtonShowDifferent ->isActive(), m_bpButtonShowEqual ->isActive(), m_bpButtonShowConflict ->isActive()); - filesOnLeftView = result.filesOnLeftView; - foldersOnLeftView = result.foldersOnLeftView; - filesOnRightView = result.filesOnRightView; - foldersOnRightView = result.foldersOnRightView; - filesizeLeftView = result.filesizeLeftView; - filesizeRightView = result.filesizeRightView; + fileCountLeft = result.fileCountLeft; + folderCountLeft = result.folderCountLeft; + bytesLeft = result.bytesLeft; + + fileCountRight = result.fileCountRight; + folderCountRight = result.folderCountRight; + bytesRight = result.bytesRight; //comparison result view buttons updateVisibility(m_bpButtonShowExcluded, result.existsExcluded); @@ -4335,29 +4335,28 @@ void MainDialog::updateGridViewData() updateVisibility(m_bpButtonShowDifferent, result.existsDifferent); } - const bool anySelectViewButtonShown = m_bpButtonShowEqual ->IsShown() || - m_bpButtonShowConflict ->IsShown() || - - m_bpButtonShowCreateLeft ->IsShown() || - m_bpButtonShowCreateRight->IsShown() || - m_bpButtonShowDeleteLeft ->IsShown() || - m_bpButtonShowDeleteRight->IsShown() || - m_bpButtonShowUpdateLeft ->IsShown() || - m_bpButtonShowUpdateRight->IsShown() || - m_bpButtonShowDoNothing ->IsShown() || + const bool anyViewButtonShown = m_bpButtonShowExcluded ->IsShown() || + m_bpButtonShowEqual ->IsShown() || + m_bpButtonShowConflict ->IsShown() || - m_bpButtonShowLeftOnly ->IsShown() || - m_bpButtonShowRightOnly ->IsShown() || - m_bpButtonShowLeftNewer ->IsShown() || - m_bpButtonShowRightNewer->IsShown() || - m_bpButtonShowDifferent ->IsShown(); + m_bpButtonShowCreateLeft ->IsShown() || + m_bpButtonShowCreateRight->IsShown() || + m_bpButtonShowDeleteLeft ->IsShown() || + m_bpButtonShowDeleteRight->IsShown() || + m_bpButtonShowUpdateLeft ->IsShown() || + m_bpButtonShowUpdateRight->IsShown() || + m_bpButtonShowDoNothing ->IsShown() || - const bool anyViewButtonShown = anySelectViewButtonShown || m_bpButtonShowExcluded->IsShown(); + m_bpButtonShowLeftOnly ->IsShown() || + m_bpButtonShowRightOnly ->IsShown() || + m_bpButtonShowLeftNewer ->IsShown() || + m_bpButtonShowRightNewer->IsShown() || + m_bpButtonShowDifferent ->IsShown(); m_staticTextViewType ->Show(anyViewButtonShown); m_bpButtonViewTypeSyncAction->Show(anyViewButtonShown); - m_staticTextSelectView ->Show(anySelectViewButtonShown); - m_bpButtonViewFilterSave ->Show(anySelectViewButtonShown); + m_staticTextSelectView ->Show(anyViewButtonShown); + m_bpButtonViewFilterSave ->Show(anyViewButtonShown); m_panelViewFilter->Layout(); @@ -4388,12 +4387,12 @@ void MainDialog::updateGridViewData() m_gridOverview->Refresh(); //update status bar information - setStatusBarFileStatistics(filesOnLeftView, - foldersOnLeftView, - filesOnRightView, - foldersOnRightView, - filesizeLeftView, - filesizeRightView); + setStatusBarFileStats(fileCountLeft, + folderCountLeft, + bytesLeft, + fileCountRight, + folderCountRight, + bytesRight); } diff --git a/FreeFileSync/Source/ui/main_dlg.h b/FreeFileSync/Source/ui/main_dlg.h index 3a66974e..7259d3d8 100755..100644 --- a/FreeFileSync/Source/ui/main_dlg.h +++ b/FreeFileSync/Source/ui/main_dlg.h @@ -70,7 +70,6 @@ private: friend class PanelMoveWindow; //configuration load/save - void setLastUsedConfig(const Zstring& cfgFilePath, const XmlGuiConfig& guiConfig) { setLastUsedConfig(std::vector<Zstring>({ cfgFilePath }), guiConfig); } void setLastUsedConfig(const std::vector<Zstring>& cfgFilePaths, const XmlGuiConfig& guiConfig); XmlGuiConfig getConfig() const; @@ -81,8 +80,8 @@ private: bool loadConfiguration(const std::vector<Zstring>& filepaths); //return true if loaded successfully - bool trySaveConfig (const Zstring* guiFilename); //return true if saved successfully - bool trySaveBatchConfig(const Zstring* batchFileToUpdate); // + bool trySaveConfig (const Zstring* guiCfgPath); //return true if saved successfully + bool trySaveBatchConfig(const Zstring* batchCfgPath); // bool saveOldConfig(); //return false on user abort void updateGlobalFilterButton(); @@ -127,10 +126,15 @@ private: const std::vector<FileSystemObject*>& selectionRight); //selection may be empty //status bar supports one of the following two states at a time: - void setStatusBarFileStatistics(size_t filesOnLeftView, size_t foldersOnLeftView, size_t filesOnRightView, size_t foldersOnRightView, uint64_t filesizeLeftView, uint64_t filesizeRightView); + void setStatusBarFileStats(size_t fileCountLeft, + size_t folderCountLeft, + uint64_t bytesLeft, + size_t fileCountRight, + size_t folderCountRight, + uint64_t bytesRight); //void setStatusBarFullText(const wxString& msg); - void flashStatusInformation(const wxString& msg); //temporarily show different status (only valid for setStatusBarFileStatistics) + void flashStatusInformation(const wxString& msg); //temporarily show different status (only valid for setStatusBarFileStats) //events void onGridButtonEventL(wxKeyEvent& event) { onGridButtonEvent(event, *m_gridMainL, true); } diff --git a/FreeFileSync/Source/ui/progress_indicator.cpp b/FreeFileSync/Source/ui/progress_indicator.cpp index 262f721b..262f721b 100755..100644 --- a/FreeFileSync/Source/ui/progress_indicator.cpp +++ b/FreeFileSync/Source/ui/progress_indicator.cpp diff --git a/FreeFileSync/Source/ui/progress_indicator.h b/FreeFileSync/Source/ui/progress_indicator.h index 5c6dae86..5c6dae86 100755..100644 --- a/FreeFileSync/Source/ui/progress_indicator.h +++ b/FreeFileSync/Source/ui/progress_indicator.h diff --git a/FreeFileSync/Source/ui/search_grid.cpp b/FreeFileSync/Source/ui/search_grid.cpp index cd03aa8d..cd03aa8d 100755..100644 --- a/FreeFileSync/Source/ui/search_grid.cpp +++ b/FreeFileSync/Source/ui/search_grid.cpp diff --git a/FreeFileSync/Source/ui/search_grid.h b/FreeFileSync/Source/ui/search_grid.h index 81560f7c..81560f7c 100755..100644 --- a/FreeFileSync/Source/ui/search_grid.h +++ b/FreeFileSync/Source/ui/search_grid.h diff --git a/FreeFileSync/Source/ui/small_dlgs.cpp b/FreeFileSync/Source/ui/small_dlgs.cpp index c37d88e2..4312a7f9 100755..100644 --- a/FreeFileSync/Source/ui/small_dlgs.cpp +++ b/FreeFileSync/Source/ui/small_dlgs.cpp @@ -38,6 +38,12 @@ + #include "abstract_folder_picker.h" + #include "../fs/concrete.h" + #include "../fs/gdrive.h" + #include "../fs/sftp.h" + #include "../fs/ftp.h" + //#include "../fs/ftp_common.h" using namespace zen; using namespace fff; @@ -170,6 +176,495 @@ void fff::showAboutDialog(wxWindow* parent) //######################################################################################## +class CloudSetupDlg : public CloudSetupDlgGenerated +{ +public: + CloudSetupDlg(wxWindow* parent, Zstring& folderPathPhrase, size_t& parallelOps, const std::wstring* parallelOpsDisabledReason); + +private: + void OnOkay (wxCommandEvent& event) override; + void OnCancel(wxCommandEvent& event) override { EndModal(ReturnSmallDlg::BUTTON_CANCEL); } + void OnClose (wxCloseEvent& event) override { EndModal(ReturnSmallDlg::BUTTON_CANCEL); } + + void OnGdriveUserAdd (wxCommandEvent& event) override; + void OnGdriveUserRemove(wxCommandEvent& event) override; + void OnGdriveUserSelect(wxCommandEvent& event) override; + void OnDetectServerChannelLimit(wxCommandEvent& event) override; + void OnToggleShowPassword(wxCommandEvent& event) override; + void OnBrowseCloudFolder (wxCommandEvent& event) override; + void OnHelpFtpPerformance(wxHyperlinkEvent& event) override { displayHelpEntry(L"ftp-setup", this); } + + void OnConnectionGdrive(wxCommandEvent& event) override { type_ = CloudType::gdrive; updateGui(); } + void OnConnectionSftp (wxCommandEvent& event) override { type_ = CloudType::sftp; updateGui(); } + void OnConnectionFtp (wxCommandEvent& event) override { type_ = CloudType::ftp; updateGui(); } + + void OnAuthPassword(wxCommandEvent& event) override { sftpAuthType_ = SftpAuthType::PASSWORD; updateGui(); } + void OnAuthKeyfile (wxCommandEvent& event) override { sftpAuthType_ = SftpAuthType::KEY_FILE; updateGui(); } + void OnAuthAgent (wxCommandEvent& event) override { sftpAuthType_ = SftpAuthType::AGENT; updateGui(); } + + void OnSelectKeyfile(wxCommandEvent& event) override; + + void updateGui(); + + //work around defunct keyboard focus on macOS (or is it wxMac?) => not needed for this dialog! + //void onLocalKeyEvent(wxKeyEvent& event); + + static bool acceptFileDrop(const std::vector<Zstring>& shellItemPaths); + void onKeyFileDropped(FileDropEvent& event); + + Zstring getFolderPathPhrase() const; + + enum class CloudType + { + gdrive, + sftp, + ftp, + }; + CloudType type_ = CloudType::gdrive; + + SftpAuthType sftpAuthType_ = SftpAuthType::PASSWORD; + + AsyncGuiQueue guiQueue_; + + //output-only parameters: + Zstring& folderPathPhraseOut_; + size_t& parallelOpsOut_; +}; + + +CloudSetupDlg::CloudSetupDlg(wxWindow* parent, Zstring& folderPathPhrase, size_t& parallelOps, const std::wstring* parallelOpsDisabledReason) : + CloudSetupDlgGenerated(parent), + folderPathPhraseOut_(folderPathPhrase), + parallelOpsOut_(parallelOps) +{ + setStandardButtonLayout(*bSizerStdButtons, StdButtons().setAffirmative(m_buttonOkay).setCancel(m_buttonCancel)); + + m_toggleBtnGdrive->SetBitmap(getResourceImage(L"google_drive")); + m_toggleBtnSftp ->SetBitmap(getTransparentPixel()); //set dummy image (can't be empty!): text-only buttons are rendered smaller on OS X! + m_toggleBtnFtp ->SetBitmap(getTransparentPixel()); // + + setRelativeFontSize(*m_toggleBtnGdrive, 1.25); + setRelativeFontSize(*m_toggleBtnSftp, 1.25); + setRelativeFontSize(*m_toggleBtnFtp, 1.25); + setRelativeFontSize(*m_staticTextGdriveUser, 1.25); + + setBitmapTextLabel(*m_buttonGdriveAddUser, getResourceImage(L"user_add" ).ConvertToImage(), m_buttonGdriveAddUser ->GetLabel()); + setBitmapTextLabel(*m_buttonGdriveRemoveUser, getResourceImage(L"user_remove").ConvertToImage(), m_buttonGdriveRemoveUser->GetLabel()); + + m_bitmapGdriveSelectedUser->SetBitmap(getResourceImage(L"user_selected")); + m_bitmapServer->SetBitmap(shrinkImage(getResourceImage(L"server").ConvertToImage(), fastFromDIP(24))); + m_bitmapCloud ->SetBitmap(getResourceImage(L"cloud")); + m_bitmapPerf ->SetBitmap(getResourceImage(L"speed")); + m_bitmapServerDir->SetBitmap(IconBuffer::genericDirIcon(IconBuffer::SIZE_SMALL)); + m_checkBoxShowPassword->SetValue(false); + + m_textCtrlServer->SetHint(_("Example:") + L" website.com 66.198.240.22"); + m_textCtrlServer->SetMinSize(wxSize(fastFromDIP(260), -1)); + + m_textCtrlPort ->SetMinSize(wxSize(fastFromDIP(60), -1)); // + m_spinCtrlConnectionCount ->SetMinSize(wxSize(fastFromDIP(70), -1)); //Hack: set size (why does wxWindow::Size() not work?) + m_spinCtrlChannelCountSftp->SetMinSize(wxSize(fastFromDIP(70), -1)); // + m_spinCtrlTimeout ->SetMinSize(wxSize(fastFromDIP(70), -1)); // + + setupFileDrop(*m_panelAuth); + m_panelAuth->Connect(EVENT_DROP_FILE, FileDropEventHandler(CloudSetupDlg::onKeyFileDropped), nullptr, this); + + m_staticTextConnectionsLabelSub->SetLabel(L"(" + _("Connections") + L")"); + + //use spacer to keep dialog height stable, no matter if key file options are visible + bSizerAuthInner->Add(0, m_panelAuth->GetSize().GetHeight()); + + wxArrayString googleUsers; + try + { + for (const Zstring& googleUser: googleListConnectedUsers()) //throw FileError + googleUsers.push_back(utfTo<wxString>(googleUser)); + } + catch (const FileError& e) + { + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e.toString())); + } + m_listBoxGdriveUsers->Append(googleUsers); + + //set default values for Google Drive: use first item of m_listBoxGdriveUsers + m_staticTextGdriveUser->SetLabel(L""); + if (!googleUsers.empty()) + { + m_listBoxGdriveUsers->SetSelection(0); + m_staticTextGdriveUser->SetLabel(googleUsers[0]); + } + + m_spinCtrlTimeout->SetValue(FtpLoginInfo().timeoutSec); + assert(FtpLoginInfo().timeoutSec == SftpLoginInfo().timeoutSec); //make sure the default values are in sync + + if (acceptsItemPathPhraseGdrive(folderPathPhrase)) + { + type_ = CloudType::gdrive; + const GdrivePath gdrivePath = getResolvedGooglePath(folderPathPhrase); //noexcept + + const int selIdx = m_listBoxGdriveUsers->FindString(utfTo<wxString>(gdrivePath.userEmail), false /*caseSensitive*/); + if (selIdx != wxNOT_FOUND) + { + m_listBoxGdriveUsers->EnsureVisible(selIdx); + m_listBoxGdriveUsers->SetSelection(selIdx); + } + else + m_listBoxGdriveUsers->DeselectAll(); + m_staticTextGdriveUser->SetLabel (utfTo<wxString>(gdrivePath.userEmail)); + m_textCtrlServerPath ->ChangeValue(utfTo<wxString>(FILE_NAME_SEPARATOR + gdrivePath.itemPath.value)); + } + else if (acceptsItemPathPhraseSftp(folderPathPhrase)) + { + type_ = CloudType::sftp; + const SftpPathInfo pi = getResolvedSftpPath(folderPathPhrase); //noexcept + + if (pi.login.port > 0) + m_textCtrlPort->ChangeValue(numberTo<wxString>(pi.login.port)); + m_textCtrlServer ->ChangeValue(utfTo<wxString>(pi.login.server)); + m_textCtrlUserName ->ChangeValue(utfTo<wxString>(pi.login.username)); + sftpAuthType_ = pi.login.authType; + m_textCtrlPasswordHidden->ChangeValue(utfTo<wxString>(pi.login.password)); + m_textCtrlKeyfilePath ->ChangeValue(utfTo<wxString>(pi.login.privateKeyFilePath)); + m_textCtrlServerPath ->ChangeValue(utfTo<wxString>(FILE_NAME_SEPARATOR + pi.afsPath.value)); + m_spinCtrlChannelCountSftp->SetValue(pi.login.traverserChannelsPerConnection); + m_spinCtrlTimeout ->SetValue(pi.login.timeoutSec); + } + else if (acceptsItemPathPhraseFtp(folderPathPhrase)) + { + type_ = CloudType::ftp; + const FtpPathInfo pi = getResolvedFtpPath(folderPathPhrase); //noexcept + + if (pi.login.port > 0) + m_textCtrlPort->ChangeValue(numberTo<wxString>(pi.login.port)); + m_textCtrlServer ->ChangeValue(utfTo<wxString>(pi.login.server)); + m_textCtrlUserName ->ChangeValue(utfTo<wxString>(pi.login.username)); + m_textCtrlPasswordHidden ->ChangeValue(utfTo<wxString>(pi.login.password)); + m_textCtrlServerPath ->ChangeValue(utfTo<wxString>(FILE_NAME_SEPARATOR + pi.afsPath.value)); + (pi.login.useSsl ? m_radioBtnEncryptSsl : m_radioBtnEncryptNone)->SetValue(true); + m_spinCtrlTimeout ->SetValue(pi.login.timeoutSec); + } + + m_spinCtrlConnectionCount->SetValue(parallelOps); + + if (parallelOpsDisabledReason) + { + //m_staticTextConnectionsLabel ->Disable(); + //m_staticTextConnectionsLabelSub->Disable(); + m_spinCtrlChannelCountSftp ->Disable(); + m_buttonChannelCountSftp ->Disable(); + m_spinCtrlConnectionCount ->Disable(); + m_staticTextConnectionCountDescr->SetLabel(*parallelOpsDisabledReason); + //m_staticTextConnectionCountDescr->SetForegroundColour(wxSystemSettings::GetColour(wxSYS_COLOUR_WINDOWTEXT)); + } + else + m_staticTextConnectionCountDescr->SetLabel(_("Recommended range:") + L" [1" + EN_DASH + L"10]"); //no spaces! + + //set up default view for dialog size calculation + bSizerGdrive->Show(false); + bSizerFtpEncrypt->Show(false); + m_textCtrlPasswordHidden->Hide(); + + GetSizer()->SetSizeHints(this); //~=Fit() + SetMinSize() + //=> works like a charm for GTK2 with window resizing problems and title bar corruption; e.g. Debian!!! + Center(); //needs to be re-applied after a dialog size change! + + updateGui(); //*after* SetSizeHints when standard dialog height has been calculated + + m_buttonOkay->SetFocus(); +} + + +void CloudSetupDlg::OnGdriveUserAdd(wxCommandEvent& event) +{ + guiQueue_.processAsync([]() -> std::variant<Zstring, FileError> + { + try + { + return googleAddUser(nullptr /*updateGui*/); //throw FileError + } + catch (const FileError& e) { return e; } + }, + [this](const std::variant<Zstring, FileError>& result) + { + if (const FileError* e = std::get_if<FileError>(&result)) + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e->toString())); + else + { + const wxString googleUser = utfTo<wxString>(std::get<Zstring>(result)); + int selIdx = m_listBoxGdriveUsers->FindString(googleUser, false /*caseSensitive*/); + if (selIdx == wxNOT_FOUND) + selIdx = m_listBoxGdriveUsers->Append(googleUser); + + m_listBoxGdriveUsers->EnsureVisible(selIdx); + m_listBoxGdriveUsers->SetSelection(selIdx); + m_staticTextGdriveUser->SetLabel(googleUser); + updateGui(); //enable remove user button + } + }); +} + + +void CloudSetupDlg::OnGdriveUserRemove(wxCommandEvent& event) +{ + const int selIdx = m_listBoxGdriveUsers->GetSelection(); + assert(selIdx != wxNOT_FOUND); + if (selIdx != wxNOT_FOUND) + try + { + const wxString googleUser = m_listBoxGdriveUsers->GetString(selIdx); + if (showConfirmationDialog(this, DialogInfoType::WARNING, PopupDialogCfg(). + setTitle(_("Confirm")). + setMainInstructions(replaceCpy(_("Do you really want to disconnect from user account %x?"), L"%x", googleUser)), + _("&Disconnect")) != ConfirmationButton::ACCEPT) + return; + + googleRemoveUser(utfTo<Zstring>(googleUser)); //throw FileError + m_listBoxGdriveUsers->Delete(selIdx); + updateGui(); //disable remove user button + //no need to also clear m_staticTextGdriveUser + } + catch (const FileError& e) + { + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e.toString())); + } +} + + +void CloudSetupDlg::OnGdriveUserSelect(wxCommandEvent& event) +{ + const int selIdx = m_listBoxGdriveUsers->GetSelection(); + assert(selIdx != wxNOT_FOUND); + if (selIdx != wxNOT_FOUND) + { + m_staticTextGdriveUser->SetLabel(m_listBoxGdriveUsers->GetString(selIdx)); + updateGui(); //enable remove user button + } +} + + +void CloudSetupDlg::OnDetectServerChannelLimit(wxCommandEvent& event) +{ + assert (type_ == CloudType::sftp); + const SftpPathInfo pi = getResolvedSftpPath(getFolderPathPhrase()); //noexcept + try + { + const int channelCountMax = getServerMaxChannelsPerConnection(pi.login); //throw FileError + m_spinCtrlChannelCountSftp->SetValue(channelCountMax); + } + catch (const FileError& e) + { + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e.toString())); + } +} + + +void CloudSetupDlg::OnToggleShowPassword(wxCommandEvent& event) +{ + assert(type_ != CloudType::gdrive); + if (m_checkBoxShowPassword->GetValue()) + m_textCtrlPasswordVisible->ChangeValue(m_textCtrlPasswordHidden->GetValue()); + else + m_textCtrlPasswordHidden->ChangeValue(m_textCtrlPasswordVisible->GetValue()); + updateGui(); +} + + +bool CloudSetupDlg::acceptFileDrop(const std::vector<Zstring>& shellItemPaths) +{ + if (shellItemPaths.empty()) + return false; + const Zstring ext = getFileExtension(shellItemPaths[0]); + return ext.empty() || equalAsciiNoCase(ext, Zstr("pem")); +} + + +void CloudSetupDlg::onKeyFileDropped(FileDropEvent& event) +{ + //assert (type_ == CloudType::SFTP); -> no big deal if false + const auto& itemPaths = event.getPaths(); + if (!itemPaths.empty()) + { + m_textCtrlKeyfilePath->ChangeValue(utfTo<wxString>(itemPaths[0])); + + sftpAuthType_ = SftpAuthType::KEY_FILE; + updateGui(); + } +} + + +void CloudSetupDlg::OnSelectKeyfile(wxCommandEvent& event) +{ + assert (type_ == CloudType::sftp && sftpAuthType_ == SftpAuthType::KEY_FILE); + wxFileDialog filePicker(this, + wxString(), + beforeLast(m_textCtrlKeyfilePath->GetValue(), utfTo<wxString>(FILE_NAME_SEPARATOR), IF_MISSING_RETURN_NONE), //default dir + wxString(), //default file + _("All files") + L" (*.*)|*" + L"|" + L"OpenSSL PEM (*.pem)|*.pem", + wxFD_OPEN); + if (filePicker.ShowModal() == wxID_OK) + m_textCtrlKeyfilePath->ChangeValue(filePicker.GetPath()); +} + + +void CloudSetupDlg::updateGui() +{ + + m_toggleBtnGdrive->SetValue(type_ == CloudType::gdrive); + m_toggleBtnSftp ->SetValue(type_ == CloudType::sftp); + m_toggleBtnFtp ->SetValue(type_ == CloudType::ftp); + + bSizerGdrive->Show(type_ == CloudType::gdrive); + bSizerServer->Show(type_ == CloudType::ftp || type_ == CloudType::sftp); + bSizerAuth ->Show(type_ == CloudType::ftp || type_ == CloudType::sftp); + + bSizerFtpEncrypt->Show(type_ == CloudType:: ftp); + bSizerSftpAuth ->Show(type_ == CloudType::sftp); + + m_staticTextKeyfile->Show(type_ == CloudType::sftp && sftpAuthType_ == SftpAuthType::KEY_FILE); + bSizerKeyFile ->Show(type_ == CloudType::sftp && sftpAuthType_ == SftpAuthType::KEY_FILE); + + m_staticTextPassword->Show(type_ == CloudType::ftp || (type_ == CloudType::sftp && sftpAuthType_ != SftpAuthType::AGENT)); + bSizerPassword ->Show(type_ == CloudType::ftp || (type_ == CloudType::sftp && sftpAuthType_ != SftpAuthType::AGENT)); + if (m_staticTextPassword->IsShown()) + { + m_textCtrlPasswordVisible->Show( m_checkBoxShowPassword->GetValue()); + m_textCtrlPasswordHidden ->Show(!m_checkBoxShowPassword->GetValue()); + } + + bSizerAccessTimeout->Show(type_ == CloudType::ftp || type_ == CloudType::sftp); + + switch (type_) + { + case CloudType::gdrive: + m_buttonGdriveRemoveUser->Enable(m_listBoxGdriveUsers->GetSelection() != wxNOT_FOUND); + break; + + case CloudType::sftp: + m_radioBtnPassword->SetValue(false); + m_radioBtnKeyfile ->SetValue(false); + m_radioBtnAgent ->SetValue(false); + + switch (sftpAuthType_) //*not* owned by GUI controls + { + case SftpAuthType::PASSWORD: + m_radioBtnPassword->SetValue(true); + m_staticTextPassword->SetLabel(_("Password:")); + break; + case SftpAuthType::KEY_FILE: + m_radioBtnKeyfile->SetValue(true); + m_staticTextPassword->SetLabel(_("Key password:")); + break; + case SftpAuthType::AGENT: + m_radioBtnAgent->SetValue(true); + break; + } + break; + + case CloudType::ftp: + m_staticTextPassword->SetLabel(_("Password:")); + break; + } + + m_staticTextChannelCountSftp->Show(type_ == CloudType::sftp); + m_spinCtrlChannelCountSftp ->Show(type_ == CloudType::sftp); + m_buttonChannelCountSftp ->Show(type_ == CloudType::sftp); + + Layout(); //needed! hidden items are not considered during resize + Refresh(); +} + + +Zstring CloudSetupDlg::getFolderPathPhrase() const +{ + switch (type_) + { + case CloudType::gdrive: + return condenseToGoogleFolderPathPhrase(utfTo<Zstring>(m_staticTextGdriveUser->GetLabel()), + utfTo<Zstring>(m_textCtrlServerPath ->GetValue())); //noexcept + + case CloudType::sftp: + { + SftpLoginInfo login; + login.server = utfTo<Zstring>(m_textCtrlServer ->GetValue()); + login.port = stringTo<int> (m_textCtrlPort ->GetValue()); //0 if empty + login.username = utfTo<Zstring>(m_textCtrlUserName->GetValue()); + login.authType = sftpAuthType_; + login.privateKeyFilePath = utfTo<Zstring>(m_textCtrlKeyfilePath->GetValue()); + login.password = utfTo<Zstring>((m_checkBoxShowPassword->GetValue() ? m_textCtrlPasswordVisible : m_textCtrlPasswordHidden)->GetValue()); + login.traverserChannelsPerConnection = m_spinCtrlChannelCountSftp->GetValue(); + login.timeoutSec = m_spinCtrlTimeout->GetValue(); + + auto serverPath = utfTo<Zstring>(m_textCtrlServerPath->GetValue()); + //clean up (messy) user input: + return condenseToSftpFolderPathPhrase(login, serverPath); //noexcept + } + + case CloudType::ftp: + { + FtpLoginInfo login; + login.server = utfTo<Zstring>(m_textCtrlServer ->GetValue()); + login.port = stringTo<int> (m_textCtrlPort ->GetValue()); //0 if empty + login.username = utfTo<Zstring>(m_textCtrlUserName->GetValue()); + login.password = utfTo<Zstring>((m_checkBoxShowPassword->GetValue() ? m_textCtrlPasswordVisible : m_textCtrlPasswordHidden)->GetValue()); + login.useSsl = m_radioBtnEncryptSsl->GetValue(); + login.timeoutSec = m_spinCtrlTimeout->GetValue(); + + auto serverPath = utfTo<Zstring>(m_textCtrlServerPath->GetValue()); + //clean up (messy) user input: + return condenseToFtpFolderPathPhrase(login, serverPath); //noexcept + } + } + assert(false); + return Zstr(""); +} + + +void CloudSetupDlg::OnBrowseCloudFolder(wxCommandEvent& event) +{ + AbstractPath folderPath = createAbstractPath(getFolderPathPhrase()); //noexcept + + try + { + //for SFTP it makes more sense to start with the home directory rather than root (which often denies access!) + if (type_ == CloudType::sftp && !AFS::getParentPath(folderPath)) + folderPath.afsPath = getSftpHomePath(getResolvedSftpPath(getFolderPathPhrase()).login); //throw FileError + } + catch (const FileError& e) + { + showNotificationDialog(this, DialogInfoType::ERROR2, PopupDialogCfg().setDetailInstructions(e.toString())); + return; + } + + if (showAbstractFolderPicker(this, folderPath) == ReturnAfsPicker::BUTTON_OKAY) + m_textCtrlServerPath->ChangeValue(utfTo<wxString>(FILE_NAME_SEPARATOR + folderPath.afsPath.value)); +} + + +void CloudSetupDlg::OnOkay(wxCommandEvent& event) +{ + //------- parameter validation (BEFORE writing output!) ------- + if (type_ == CloudType::sftp && sftpAuthType_ == SftpAuthType::KEY_FILE) + if (trimCpy(m_textCtrlKeyfilePath->GetValue()).empty()) + { + showNotificationDialog(this, DialogInfoType::INFO, PopupDialogCfg().setMainInstructions(_("Please enter a file path."))); + //don't show error icon to follow "Windows' encouraging tone" + m_textCtrlKeyfilePath->SetFocus(); + return; + } + //------------------------------------------------------------- + + folderPathPhraseOut_ = getFolderPathPhrase(); + parallelOpsOut_ = m_spinCtrlConnectionCount->GetValue(); + + EndModal(ReturnSmallDlg::BUTTON_OKAY); +} + + +ReturnSmallDlg::ButtonPressed fff::showCloudSetupDialog(wxWindow* parent, Zstring& folderPathPhrase, size_t& parallelOps, const std::wstring* parallelOpsDisabledReason) +{ + CloudSetupDlg setupDlg(parent, folderPathPhrase, parallelOps, parallelOpsDisabledReason); + return static_cast<ReturnSmallDlg::ButtonPressed>(setupDlg.ShowModal()); +} //######################################################################################## @@ -221,7 +716,7 @@ CopyToDialog::CopyToDialog(wxWindow* parent, m_bitmapCopyTo->SetBitmap(getResourceImage(L"copy_to")); - targetFolder = std::make_unique<FolderSelector>(*this, *m_buttonSelectTargetFolder, *m_bpButtonSelectAltTargetFolder, *m_targetFolderPath, nullptr /*staticText*/, nullptr /*wxWindow*/, + targetFolder = std::make_unique<FolderSelector>(this, *this, *m_buttonSelectTargetFolder, *m_bpButtonSelectAltTargetFolder, *m_targetFolderPath, nullptr /*staticText*/, nullptr /*wxWindow*/, nullptr /*droppedPathsFilter*/, [](const Zstring& folderPathPhrase) { return 1; } /*getDeviceParallelOps*/, nullptr /*setDeviceParallelOps*/); @@ -587,7 +1082,7 @@ OptionsDlg::OptionsDlg(wxWindow* parent, XmlGlobalSettings& globalSettings) : //setMainInstructionFont(*m_staticTextHeader); m_gridCustomCommand->SetTabBehaviour(wxGrid::Tab_Leave); - m_bitmapLogFile->SetBitmap(getResourceImage(L"log_file_sicon")); + m_bitmapLogFile->SetBitmap(shrinkImage(getResourceImage(L"log_file").ConvertToImage(), fastFromDIP(20))); m_spinCtrlLogFilesMaxAge->SetMinSize(wxSize(fastFromDIP(70), -1)); //Hack: set size (why does wxWindow::Size() not work?) m_hyperlinkLogFolder->SetLabel(utfTo<wxString>(getDefaultLogFolderPath())); setRelativeFontSize(*m_hyperlinkLogFolder, 1.2); @@ -665,7 +1160,8 @@ void OptionsDlg::updateGui() warnDlgs_ != defaultCfg_.warnDlgs || autoCloseProgressDialog_ != defaultCfg_.autoCloseProgressDialog; - setBitmapTextLabel(*m_buttonResetDialogs, getResourceImage(L"reset_dialogs").ConvertToImage(), haveHiddenDialogs ? _("Show hidden dialogs again") : _("All dialogs shown")); + setBitmapTextLabel(*m_buttonResetDialogs, shrinkImage(getResourceImage(L"msg_warning").ConvertToImage(), fastFromDIP(20)), + haveHiddenDialogs ? _("Show hidden dialogs again") : _("All dialogs shown")); Layout(); m_buttonResetDialogs->Enable(haveHiddenDialogs); diff --git a/FreeFileSync/Source/ui/small_dlgs.h b/FreeFileSync/Source/ui/small_dlgs.h index c03415f1..9083874c 100755..100644 --- a/FreeFileSync/Source/ui/small_dlgs.h +++ b/FreeFileSync/Source/ui/small_dlgs.h @@ -53,6 +53,8 @@ ReturnSmallDlg::ButtonPressed showSelectTimespanDlg(wxWindow* parent, time_t& ti ReturnSmallDlg::ButtonPressed showCfgHighlightDlg(wxWindow* parent, int& cfgHistSyncOverdueDays); +ReturnSmallDlg::ButtonPressed showCloudSetupDialog(wxWindow* parent, Zstring& folderPathPhrase, + size_t& parallelOps, const std::wstring* parallelOpsDisabledReason /*optional: disable control + show text*/); enum class ReturnActivationDlg { diff --git a/FreeFileSync/Source/ui/sorting.h b/FreeFileSync/Source/ui/sorting.h index e9432907..e9432907 100755..100644 --- a/FreeFileSync/Source/ui/sorting.h +++ b/FreeFileSync/Source/ui/sorting.h diff --git a/FreeFileSync/Source/ui/sync_cfg.cpp b/FreeFileSync/Source/ui/sync_cfg.cpp index d0876de2..8a2add95 100755..100644 --- a/FreeFileSync/Source/ui/sync_cfg.cpp +++ b/FreeFileSync/Source/ui/sync_cfg.cpp @@ -252,10 +252,10 @@ setDeviceParallelOps_([this](const Zstring& folderPathPhrase, size_t parallelOps setDeviceParallelOps(globalPairCfg_.miscCfg.deviceParallelOps, folderPathPhrase, parallelOps); }), -versioningFolder_(*m_panelVersioning, *m_buttonSelectVersioningFolder, *m_bpButtonSelectVersioningAltFolder, *m_versioningFolderPath, +versioningFolder_(this, *m_panelVersioning, *m_buttonSelectVersioningFolder, *m_bpButtonSelectVersioningAltFolder, *m_versioningFolderPath, nullptr /*staticText*/, nullptr /*dropWindow2*/, nullptr /*droppedPathsFilter*/, getDeviceParallelOps_, setDeviceParallelOps_), -logfileDir_(*m_panelLogfile, *m_buttonSelectLogFolder, *m_bpButtonSelectAltLogFolder, *m_logFolderPath, +logfileDir_(this, *m_panelLogfile, *m_buttonSelectLogFolder, *m_bpButtonSelectAltLogFolder, *m_logFolderPath, nullptr /*staticText*/, nullptr /*dropWindow2*/, nullptr /*droppedPathsFilter*/, getDeviceParallelOps_, setDeviceParallelOps_), globalPairCfgOut_(globalPairCfg), @@ -263,7 +263,7 @@ localPairCfgOut_(localPairConfig), globalPairCfg_(globalPairCfg), localPairCfg_(localPairConfig), showMultipleCfgs_(showMultipleCfgs), -perfPanelActive_(true), +perfPanelActive_(false), commandHistItemsMax_(commandHistItemsMax) { setStandardButtonLayout(*bSizerStdButtons, StdButtons().setAffirmative(m_buttonOkay).setCancel(m_buttonCancel)); @@ -1267,9 +1267,10 @@ void ConfigDialog::updateMiscGui() m_panelComparisonSettings->Layout(); //showing "retry count" can affect bSizerPerformance! //---------------------------------------------------------------------------- - m_bitmapLogFile->SetBitmap(m_checkBoxSaveLog->GetValue() ? getResourceImage(L"log_file_sicon") : greyScale(getResourceImage(L"log_file_sicon"))); + m_bitmapLogFile->SetBitmap(shrinkImage(getResourceImage(L"log_file").ConvertToImage(), fastFromDIP(20))); m_logFolderPath ->Enable(m_checkBoxSaveLog->GetValue()); // m_buttonSelectLogFolder ->Show(m_checkBoxSaveLog->GetValue()); //enabled status is *not* directly dependent from resolved config! (but transitively) + m_bpButtonSelectAltLogFolder->Show(m_checkBoxSaveLog->GetValue()); // m_panelSyncSettings->Layout(); //after showing/hiding m_buttonSelectLogFolder } diff --git a/FreeFileSync/Source/ui/sync_cfg.h b/FreeFileSync/Source/ui/sync_cfg.h index 34c4acf2..34c4acf2 100755..100644 --- a/FreeFileSync/Source/ui/sync_cfg.h +++ b/FreeFileSync/Source/ui/sync_cfg.h diff --git a/FreeFileSync/Source/ui/taskbar.cpp b/FreeFileSync/Source/ui/taskbar.cpp index c4bc7bec..c4bc7bec 100755..100644 --- a/FreeFileSync/Source/ui/taskbar.cpp +++ b/FreeFileSync/Source/ui/taskbar.cpp diff --git a/FreeFileSync/Source/ui/taskbar.h b/FreeFileSync/Source/ui/taskbar.h index 667d8afd..667d8afd 100755..100644 --- a/FreeFileSync/Source/ui/taskbar.h +++ b/FreeFileSync/Source/ui/taskbar.h diff --git a/FreeFileSync/Source/ui/tray_icon.cpp b/FreeFileSync/Source/ui/tray_icon.cpp index 22d91f66..22d91f66 100755..100644 --- a/FreeFileSync/Source/ui/tray_icon.cpp +++ b/FreeFileSync/Source/ui/tray_icon.cpp diff --git a/FreeFileSync/Source/ui/tray_icon.h b/FreeFileSync/Source/ui/tray_icon.h index d0aeaf78..d0aeaf78 100755..100644 --- a/FreeFileSync/Source/ui/tray_icon.h +++ b/FreeFileSync/Source/ui/tray_icon.h diff --git a/FreeFileSync/Source/ui/tree_grid.cpp b/FreeFileSync/Source/ui/tree_grid.cpp index 8d5b71a1..2525ace2 100755..100644 --- a/FreeFileSync/Source/ui/tree_grid.cpp +++ b/FreeFileSync/Source/ui/tree_grid.cpp @@ -16,6 +16,7 @@ #include <wx+/dc.h> #include <wx+/context_menu.h> #include <wx+/image_resources.h> +#include <wx+/image_tools.h> #include "../base/icon_buffer.h" using namespace zen; @@ -703,7 +704,7 @@ public: widthNodeIcon_(IconBuffer::getSize(IconBuffer::SIZE_SMALL)), widthLevelStep_(widthNodeIcon_), widthNodeStatus_(getResourceImage(L"node_expanded").GetWidth()), - rootBmp_(getResourceImage(L"rootFolder").ConvertToImage().Scale(widthNodeIcon_, widthNodeIcon_, wxIMAGE_QUALITY_BILINEAR)), //looks sharper than wxIMAGE_QUALITY_HIGH! + rootBmp_(shrinkImage(getResourceImage(L"root_folder").ConvertToImage(), widthNodeIcon_)), grid_(grid) { grid.getMainWin().Connect(wxEVT_KEY_DOWN, wxKeyEventHandler(GridDataTree::onKeyDown), nullptr, this); diff --git a/FreeFileSync/Source/ui/tree_grid.h b/FreeFileSync/Source/ui/tree_grid.h index 027271af..027271af 100755..100644 --- a/FreeFileSync/Source/ui/tree_grid.h +++ b/FreeFileSync/Source/ui/tree_grid.h diff --git a/FreeFileSync/Source/ui/tree_grid_attr.h b/FreeFileSync/Source/ui/tree_grid_attr.h index aea8130a..aea8130a 100755..100644 --- a/FreeFileSync/Source/ui/tree_grid_attr.h +++ b/FreeFileSync/Source/ui/tree_grid_attr.h diff --git a/FreeFileSync/Source/ui/triple_splitter.cpp b/FreeFileSync/Source/ui/triple_splitter.cpp index 0797b807..0797b807 100755..100644 --- a/FreeFileSync/Source/ui/triple_splitter.cpp +++ b/FreeFileSync/Source/ui/triple_splitter.cpp diff --git a/FreeFileSync/Source/ui/triple_splitter.h b/FreeFileSync/Source/ui/triple_splitter.h index ea7974fe..ea7974fe 100755..100644 --- a/FreeFileSync/Source/ui/triple_splitter.h +++ b/FreeFileSync/Source/ui/triple_splitter.h diff --git a/FreeFileSync/Source/ui/version_check.cpp b/FreeFileSync/Source/ui/version_check.cpp index 9ff430df..2bb5f828 100755..100644 --- a/FreeFileSync/Source/ui/version_check.cpp +++ b/FreeFileSync/Source/ui/version_check.cpp @@ -5,9 +5,12 @@ // ***************************************************************************** #include "version_check.h" +#include <ctime> +#include <zen/crc.h> #include <zen/string_tools.h> #include <zen/i18n.h> #include <zen/utf.h> +#include <zen/file_access.h> #include <zen/scope_guard.h> #include <zen/build_info.h> #include <zen/basic_math.h> @@ -17,8 +20,8 @@ #include <wx+/popup_dlg.h> #include <wx+/image_resources.h> #include "../base/ffs_paths.h" +#include "../version/version.h" #include "small_dlgs.h" -#include "version_check_impl.h" @@ -30,6 +33,50 @@ namespace { const Zchar ffsUpdateCheckUserAgent[] = Zstr("FFS-Update-Check"); + +time_t getVersionCheckInactiveId() +{ + //use current version to calculate a changing number for the inactive state near UTC begin, in order to always check for updates after installing a new version + //=> interpret version as 11-based *unique* number (this breaks lexicographical version ordering, but that's irrelevant!) + int id = 0; + const char* first = ffsVersion; + const char* last = first + zen::strLength(ffsVersion); + std::for_each(first, last, [&](char c) + { + id *= 11; + if ('0' <= c && c <= '9') + id += c - '0'; + else + { + assert(c == FFS_VERSION_SEPARATOR); + id += 10; + } + }); + assert(0 < id && id < 3600 * 24 * 365); //as long as value is within a year after UTC begin (1970) there's no risk to clash with *current* time + return id; +} + + + + +time_t getVersionCheckCurrentTime() +{ + time_t now = std::time(nullptr); + return now; +} +} + + +bool fff::shouldRunAutomaticUpdateCheck(time_t lastUpdateCheck) +{ + if (lastUpdateCheck == getVersionCheckInactiveId()) + return false; + + const time_t now = std::time(nullptr); + return numeric::dist(now, lastUpdateCheck) >= 7 * 24 * 3600; //check weekly +} + + std::wstring getIso639Language() { assert(runningMainThread()); //this function is not thread-safe, consider wxWidgets usage @@ -47,6 +94,8 @@ std::wstring getIso639Language() } +namespace +{ std::wstring getIso3166Country() { assert(runningMainThread()); //this function is not thread-safe, consider wxWidgets usage diff --git a/FreeFileSync/Source/ui/version_check.h b/FreeFileSync/Source/ui/version_check.h index 013849e4..013849e4 100755..100644 --- a/FreeFileSync/Source/ui/version_check.h +++ b/FreeFileSync/Source/ui/version_check.h diff --git a/FreeFileSync/Source/version/version.h b/FreeFileSync/Source/version/version.h index c5482d96..f3b88a49 100755..100644 --- a/FreeFileSync/Source/version/version.h +++ b/FreeFileSync/Source/version/version.h @@ -3,7 +3,7 @@ namespace fff { -const char ffsVersion[] = "10.8"; //internal linkage! +const char ffsVersion[] = "10.9"; //internal linkage! const char FFS_VERSION_SEPARATOR = '.'; } diff --git a/wx+/app_main.h b/wx+/app_main.h index 8d2a6eeb..8d2a6eeb 100755..100644 --- a/wx+/app_main.h +++ b/wx+/app_main.h diff --git a/wx+/async_task.h b/wx+/async_task.h index 074f5337..04eac61c 100755..100644 --- a/wx+/async_task.h +++ b/wx+/async_task.h @@ -115,7 +115,7 @@ private: class AsyncGuiQueue : private wxEvtHandler { public: - AsyncGuiQueue() { timer_.Connect(wxEVT_TIMER, wxEventHandler(AsyncGuiQueue::onTimerEvent), nullptr, this); } + AsyncGuiQueue(int pollingMs = 50) : pollingMs_(pollingMs) { timer_.Connect(wxEVT_TIMER, wxEventHandler(AsyncGuiQueue::onTimerEvent), nullptr, this); } template <class Fun, class Fun2> void processAsync(Fun&& evalAsync, Fun2&& evalOnGui) @@ -123,7 +123,7 @@ public: asyncTasks_.add(std::forward<Fun >(evalAsync), std::forward<Fun2>(evalOnGui)); if (!timer_.IsRunning()) - timer_.Start(50 /*unit: [ms]*/); + timer_.Start(pollingMs_ /*unit: [ms]*/); } private: @@ -134,6 +134,7 @@ private: timer_.Stop(); } + const int pollingMs_; impl::AsyncTasks asyncTasks_; wxTimer timer_; //don't use wxWidgets' idle handling => repeated idle requests/consumption hogs 100% cpu! }; diff --git a/wx+/bitmap_button.h b/wx+/bitmap_button.h index 738a62ff..bcbc7328 100755..100644 --- a/wx+/bitmap_button.h +++ b/wx+/bitmap_button.h @@ -56,8 +56,8 @@ void setBitmapTextLabel(wxBitmapButton& btn, const wxImage& img, const wxString& wxImage dynImage = createImageFromText(text, btn.GetFont(), btn.GetForegroundColour()); if (img.IsOk()) dynImage = btn.GetLayoutDirection() != wxLayout_RightToLeft ? - stackImages(img, dynImage, ImageStackLayout::HORIZONTAL, ImageStackAlignment::CENTER, gap) : - stackImages(dynImage, img, ImageStackLayout::HORIZONTAL, ImageStackAlignment::CENTER, gap); + stackImages(img, dynImage, ImageStackLayout::HORIZONTAL, ImageStackAlignment::CENTER, gap) : + stackImages(dynImage, img, ImageStackLayout::HORIZONTAL, ImageStackAlignment::CENTER, gap); //SetMinSize() instead of SetSize() is needed here for wxWindows layout determination to work corretly const int defaultHeight = wxButton::GetDefaultSize().GetHeight(); diff --git a/wx+/choice_enum.h b/wx+/choice_enum.h index f2c93927..f2c93927 100755..100644 --- a/wx+/choice_enum.h +++ b/wx+/choice_enum.h diff --git a/wx+/context_menu.h b/wx+/context_menu.h index d856db03..d856db03 100755..100644 --- a/wx+/context_menu.h +++ b/wx+/context_menu.h @@ -52,9 +52,9 @@ inline int fastFromDIP(int d) //like wxWindow::FromDIP (but tied to primary monitor and buffered) { -#ifdef wxHAVE_DPI_INDEPENDENT_PIXELS //pulled from wx/window.h +#ifdef wxHAVE_DPI_INDEPENDENT_PIXELS //pulled from wx/window.h: https://github.com/wxWidgets/wxWidgets/blob/master/include/wx/window.h#L2029 return d; //e.g. macOS, GTK3 -#else +#else //https://github.com/wxWidgets/wxWidgets/blob/master/src/common/wincmn.cpp#L2865 assert(wxTheApp); //only call after wxWidgets was initalized! static const int dpiY = wxScreenDC().GetPPI().y; //perf: buffering for calls to ::GetDeviceCaps() needed!? const int defaultDpi = 96; diff --git a/wx+/file_drop.cpp b/wx+/file_drop.cpp index 65d5d861..65d5d861 100755..100644 --- a/wx+/file_drop.cpp +++ b/wx+/file_drop.cpp diff --git a/wx+/file_drop.h b/wx+/file_drop.h index ee5393b7..ee5393b7 100755..100644 --- a/wx+/file_drop.h +++ b/wx+/file_drop.h diff --git a/wx+/focus.h b/wx+/focus.h index e2daef79..e2daef79 100755..100644 --- a/wx+/focus.h +++ b/wx+/focus.h diff --git a/wx+/font_size.h b/wx+/font_size.h index 2f2d377c..2f2d377c 100755..100644 --- a/wx+/font_size.h +++ b/wx+/font_size.h diff --git a/wx+/graph.cpp b/wx+/graph.cpp index 9cacd1bf..9cacd1bf 100755..100644 --- a/wx+/graph.cpp +++ b/wx+/graph.cpp diff --git a/wx+/graph.h b/wx+/graph.h index f1ae5d5a..f1ae5d5a 100755..100644 --- a/wx+/graph.h +++ b/wx+/graph.h diff --git a/wx+/grid.cpp b/wx+/grid.cpp index 3c19c246..3c19c246 100755..100644 --- a/wx+/grid.cpp +++ b/wx+/grid.cpp diff --git a/wx+/grid.h b/wx+/grid.h index 102396c3..102396c3 100755..100644 --- a/wx+/grid.h +++ b/wx+/grid.h diff --git a/wx+/image_holder.h b/wx+/image_holder.h index b11ae451..b11ae451 100755..100644 --- a/wx+/image_holder.h +++ b/wx+/image_holder.h diff --git a/wx+/image_resources.cpp b/wx+/image_resources.cpp index 5bc8006f..5bc8006f 100755..100644 --- a/wx+/image_resources.cpp +++ b/wx+/image_resources.cpp diff --git a/wx+/image_resources.h b/wx+/image_resources.h index 5ea56679..5ea56679 100755..100644 --- a/wx+/image_resources.h +++ b/wx+/image_resources.h diff --git a/wx+/image_tools.cpp b/wx+/image_tools.cpp index b314e801..b314e801 100755..100644 --- a/wx+/image_tools.cpp +++ b/wx+/image_tools.cpp diff --git a/wx+/image_tools.h b/wx+/image_tools.h index bef4cb67..e1a6953c 100755..100644 --- a/wx+/image_tools.h +++ b/wx+/image_tools.h @@ -52,6 +52,8 @@ void convertToVanillaImage(wxImage& img); //add alpha channel if missing + remov //wxColor hsvColor(double h, double s, double v); //h within [0, 360), s, v within [0, 1] +wxImage shrinkImage(const wxImage& img, int requestedSize); + inline wxImage getTransparentPixel() @@ -179,6 +181,17 @@ bool isEqual(const wxBitmap& lhs, const wxBitmap& rhs) return true; } + +inline +wxImage shrinkImage(const wxImage& img, int requestedSize) +{ + const int maxExtent = std::max(img.GetWidth(), img.GetHeight()); + assert(requestedSize <= maxExtent); + return img.Scale(img.GetWidth () * requestedSize / maxExtent, + img.GetHeight() * requestedSize / maxExtent, wxIMAGE_QUALITY_BILINEAR); //looks sharper than wxIMAGE_QUALITY_HIGH! +} + + /* inline wxColor gradient(const wxColor& from, const wxColor& to, double fraction) diff --git a/wx+/no_flicker.h b/wx+/no_flicker.h index 03969c00..03969c00 100755..100644 --- a/wx+/no_flicker.h +++ b/wx+/no_flicker.h diff --git a/wx+/popup_dlg.cpp b/wx+/popup_dlg.cpp index 689b364e..689b364e 100755..100644 --- a/wx+/popup_dlg.cpp +++ b/wx+/popup_dlg.cpp diff --git a/wx+/popup_dlg.h b/wx+/popup_dlg.h index 2aedf9b8..2aedf9b8 100755..100644 --- a/wx+/popup_dlg.h +++ b/wx+/popup_dlg.h diff --git a/wx+/popup_dlg_generated.cpp b/wx+/popup_dlg_generated.cpp index 3e490757..3e490757 100755..100644 --- a/wx+/popup_dlg_generated.cpp +++ b/wx+/popup_dlg_generated.cpp diff --git a/wx+/popup_dlg_generated.h b/wx+/popup_dlg_generated.h index 9d9bc3f8..9d9bc3f8 100755..100644 --- a/wx+/popup_dlg_generated.h +++ b/wx+/popup_dlg_generated.h diff --git a/wx+/rtl.h b/wx+/rtl.h index 26380f9d..26380f9d 100755..100644 --- a/wx+/rtl.h +++ b/wx+/rtl.h diff --git a/wx+/std_button_layout.h b/wx+/std_button_layout.h index cf0152de..cf0152de 100755..100644 --- a/wx+/std_button_layout.h +++ b/wx+/std_button_layout.h diff --git a/wx+/toggle_button.h b/wx+/toggle_button.h index f61c3857..f61c3857 100755..100644 --- a/wx+/toggle_button.h +++ b/wx+/toggle_button.h diff --git a/wx+/tooltip.cpp b/wx+/tooltip.cpp index a7bf85fe..a7bf85fe 100755..100644 --- a/wx+/tooltip.cpp +++ b/wx+/tooltip.cpp diff --git a/wx+/tooltip.h b/wx+/tooltip.h index d74beb9d..d74beb9d 100755..100644 --- a/wx+/tooltip.h +++ b/wx+/tooltip.h diff --git a/xBRZ/src/xbrz.cpp b/xBRZ/src/xbrz.cpp index f37d1a01..f37d1a01 100755..100644 --- a/xBRZ/src/xbrz.cpp +++ b/xBRZ/src/xbrz.cpp diff --git a/xBRZ/src/xbrz.h b/xBRZ/src/xbrz.h index f7f7169a..f7f7169a 100755..100644 --- a/xBRZ/src/xbrz.h +++ b/xBRZ/src/xbrz.h diff --git a/xBRZ/src/xbrz_config.h b/xBRZ/src/xbrz_config.h index fcfda99a..fcfda99a 100755..100644 --- a/xBRZ/src/xbrz_config.h +++ b/xBRZ/src/xbrz_config.h diff --git a/xBRZ/src/xbrz_tools.h b/xBRZ/src/xbrz_tools.h index aaa2b769..aaa2b769 100755..100644 --- a/xBRZ/src/xbrz_tools.h +++ b/xBRZ/src/xbrz_tools.h diff --git a/zen/base64.h b/zen/base64.h new file mode 100644 index 00000000..54a0a98b --- /dev/null +++ b/zen/base64.h @@ -0,0 +1,175 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef BASE64_H_08473021856321840873021487213453214 +#define BASE64_H_08473021856321840873021487213453214 + +#include <iterator> +#include "type_traits.h" + + +namespace zen +{ +//http://en.wikipedia.org/wiki/Base64 +/* +Usage: + const std::string input = "Sample text"; + std::string output; + zen::encodeBase64(input.begin(), input.end(), std::back_inserter(output)); + //output contains "U2FtcGxlIHRleHQ=" +*/ +template <class InputIterator, class OutputIterator> +OutputIterator encodeBase64(InputIterator first, InputIterator last, OutputIterator result); //nothrow! + +template <class InputIterator, class OutputIterator> +OutputIterator decodeBase64(InputIterator first, InputIterator last, OutputIterator result); //nothrow! + +std::string stringEncodeBase64(const std::string& str); +std::string stringDecodeBase64(const std::string& str); + + + + + + + + + + +//------------------------- implementation ------------------------------- +namespace impl +{ +//64 chars for base64 encoding + padding char +constexpr char ENCODING_MIME[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="; +constexpr signed char DECODING_MIME[] = +{ + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 62, -1, -1, -1, 63, + 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, -1, 64, -1, -1, + -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, + 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, -1, -1, -1, -1, -1, + -1, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, + 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, -1, -1, -1, -1, -1 + }; +const unsigned char INDEX_PAD = 64; //"=" +} + + +template <class InputIterator, class OutputIterator> inline +OutputIterator encodeBase64(InputIterator first, InputIterator last, OutputIterator result) +{ + using namespace impl; + static_assert(sizeof(typename std::iterator_traits<InputIterator>::value_type) == 1); + static_assert(arraySize(ENCODING_MIME) == 64 + 1 + 1); + static_assert(arrayAccumulate<int>(ENCODING_MIME) + INDEX_PAD == 5602); + + while (first != last) + { + const unsigned char a = static_cast<unsigned char>(*first++); + *result++ = ENCODING_MIME[a >> 2]; + + if (first == last) + { + *result++ = ENCODING_MIME[((a & 0x3) << 4)]; + *result++ = ENCODING_MIME[INDEX_PAD]; + *result++ = ENCODING_MIME[INDEX_PAD]; + break; + } + const unsigned char b = static_cast<unsigned char>(*first++); + *result++ = ENCODING_MIME[((a & 0x3) << 4) | (b >> 4)]; + + if (first == last) + { + *result++ = ENCODING_MIME[((b & 0xf) << 2)]; + *result++ = ENCODING_MIME[INDEX_PAD]; + break; + } + const unsigned char c = static_cast<unsigned char>(*first++); + *result++ = ENCODING_MIME[((b & 0xf) << 2) | (c >> 6)]; + *result++ = ENCODING_MIME[c & 0x3f]; + } + return result; +} + + +template <class InputIterator, class OutputIterator> inline +OutputIterator decodeBase64(InputIterator first, InputIterator last, OutputIterator result) +{ + using namespace impl; + static_assert(sizeof(typename std::iterator_traits<InputIterator>::value_type) == 1); + static_assert(arraySize(DECODING_MIME) == 128); + static_assert(arrayAccumulate<int>(DECODING_MIME) + INDEX_PAD == 2081); + + const unsigned char INDEX_END = INDEX_PAD + 1; + + auto readIndex = [&]() -> unsigned char //return index within [0, 64] or INDEX_END if end of input + { + for (;;) + { + if (first == last) + return INDEX_END; + + const unsigned char ch = static_cast<unsigned char>(*first++); + if (ch < 128) //we're in lower ASCII table half + { + const int index = DECODING_MIME[ch]; + if (0 <= index && index <= static_cast<int>(INDEX_PAD)) //skip all unknown characters (including carriage return, line-break, tab) + return static_cast<unsigned char>(index); + } + } + }; + + for (;;) + { + const unsigned char index1 = readIndex(); + const unsigned char index2 = readIndex(); + if (index1 >= INDEX_PAD || index2 >= INDEX_PAD) + { + assert(index1 == INDEX_END && index2 == INDEX_END); + break; + } + *result++ = static_cast<char>((index1 << 2) | (index2 >> 4)); + + const unsigned char index3 = readIndex(); + if (index3 >= INDEX_PAD) //padding + { + assert(index3 == INDEX_PAD); + break; + } + *result++ = static_cast<char>(((index2 & 0xf) << 4) | (index3 >> 2)); + + const unsigned char index4 = readIndex(); + if (index4 >= INDEX_PAD) //padding + { + assert(index4 == INDEX_PAD); + break; + } + *result++ = static_cast<char>(((index3 & 0x3) << 6) | index4); + } + return result; +} + + +inline +std::string stringEncodeBase64(const std::string& str) +{ + std::string out; + encodeBase64(str.begin(), str.end(), std::back_inserter(out)); + return out; +} + + +inline +std::string stringDecodeBase64(const std::string& str) +{ + std::string out; + decodeBase64(str.begin(), str.end(), std::back_inserter(out)); + return out; +} +} + +#endif //BASE64_H_08473021856321840873021487213453214 diff --git a/zen/basic_math.h b/zen/basic_math.h index 75f5d3b8..75f5d3b8 100755..100644 --- a/zen/basic_math.h +++ b/zen/basic_math.h diff --git a/zen/build_info.h b/zen/build_info.h index e80f3721..e80f3721 100755..100644 --- a/zen/build_info.h +++ b/zen/build_info.h diff --git a/zen/crc.h b/zen/crc.h index df460a03..df460a03 100755..100644 --- a/zen/crc.h +++ b/zen/crc.h diff --git a/zen/dir_watcher.cpp b/zen/dir_watcher.cpp index f5ed0488..f5ed0488 100755..100644 --- a/zen/dir_watcher.cpp +++ b/zen/dir_watcher.cpp diff --git a/zen/dir_watcher.h b/zen/dir_watcher.h index f552e2b2..f552e2b2 100755..100644 --- a/zen/dir_watcher.h +++ b/zen/dir_watcher.h diff --git a/zen/error_log.h b/zen/error_log.h index 4a3f5f2c..4a3f5f2c 100755..100644 --- a/zen/error_log.h +++ b/zen/error_log.h diff --git a/zen/file_access.cpp b/zen/file_access.cpp index d257b7c2..a60eccb2 100755..100644 --- a/zen/file_access.cpp +++ b/zen/file_access.cpp @@ -60,7 +60,7 @@ std::optional<PathComponents> zen::parsePathComponents(const Zstring& itemPath) if (startsWith(itemPath, std::string("/media/") + username + "/")) pc = doParse(4 /*sepCountVolumeRoot*/, false /*rootWithSep*/); - if (!pc && startsWith(itemPath, "/run/media/")) //Centos, Suse: e.g. /run/media/zenju/DEVICE_NAME + if (!pc && startsWith(itemPath, "/run/media/")) //CentOS, Suse: e.g. /run/media/zenju/DEVICE_NAME if (const char* username = ::getenv("USER")) if (startsWith(itemPath, std::string("/run/media/") + username + "/")) pc = doParse(5 /*sepCountVolumeRoot*/, false /*rootWithSep*/); @@ -131,8 +131,8 @@ std::optional<ItemType> zen::itemStillExists(const Zstring& itemPath) //throw Fi try { traverseFolder(*parentPath, - [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, - [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, + [&](const FileInfo& fi) { if (fi.itemName == itemName) throw ItemType::FILE; }, + [&](const FolderInfo& fi) { if (fi.itemName == itemName) throw ItemType::FOLDER; }, [&](const SymlinkInfo& si) { if (si.itemName == itemName) throw ItemType::SYMLINK; }, [](const std::wstring& errorMsg) { throw FileError(errorMsg); }); } @@ -190,18 +190,18 @@ FileDetails zen::getFileDetails(const Zstring& itemPath) //throw FileError { makeUnsigned(fileInfo.st_size), fileInfo.st_mtime, - fileInfo.st_dev, - //FileIndex fileIndex = fileInfo.st_ino; + fileInfo.st_dev, + //FileIndex fileIndex = fileInfo.st_ino; }; } Zstring zen::getTempFolderPath() //throw FileError { - const char* buf = ::getenv("TMPDIR"); //no extended error reporting - if (!buf) - throw FileError(_("Cannot get process information."), L"getenv: TMPDIR not found."); - return buf; + if (const char* buf = ::getenv("TMPDIR")) //no extended error reporting + return buf; + + return P_tmpdir; //usually resolves to "/tmp" } @@ -263,8 +263,8 @@ void removeDirectoryImpl(const Zstring& folderPath) //throw FileError //get all files and directories from current directory (WITHOUT subdirectories!) traverseFolder(folderPath, - [&](const FileInfo& fi) { filePaths .push_back(fi.fullPath); }, - [&](const FolderInfo& fi) { folderPaths .push_back(fi.fullPath); }, //defer recursion => save stack space and allow deletion of extremely deep hierarchies! + [&](const FileInfo& fi) { filePaths.push_back(fi.fullPath); }, + [&](const FolderInfo& fi) { folderPaths.push_back(fi.fullPath); }, //defer recursion => save stack space and allow deletion of extremely deep hierarchies! [&](const SymlinkInfo& si) { symlinkPaths.push_back(si.fullPath); }, [](const std::wstring& errorMsg) { throw FileError(errorMsg); }); @@ -334,7 +334,7 @@ void moveAndRenameFileSub(const Zstring& pathSource, const Zstring& pathTarget, infoSrc.st_ino != infoTrg.st_ino) throwException(EEXIST); //that's what we're really here for //else: continue with a rename in case - //caveat: if we have a hardlink referenced by two different paths, the source one will be unlinked => fine, but not exactly a "rename"... + //caveat: if we have a hardlink referenced by two different paths, the source one will be unlinked => fine, but not exactly a "rename"... } //else: not existing or access error (hopefully ::rename will also fail!) } @@ -574,7 +574,6 @@ void zen::createDirectoryIfMissingRecursion(const Zstring& dirPath) //throw File return; //already existing => possible, if createDirectoryIfMissingRecursion() is run in parallel } catch (FileError&) {} //not yet existing or access error - //catch (const FileError& e2) { throw FileError(e.toString(), e2.toString()); } -> details needed??? throw; } diff --git a/zen/file_access.h b/zen/file_access.h index d981dcc3..e06c64ac 100755..100644 --- a/zen/file_access.h +++ b/zen/file_access.h @@ -102,7 +102,7 @@ struct FileCopyResult FileCopyResult copyNewFile(const Zstring& sourceFile, const Zstring& targetFile, bool copyFilePermissions, //throw FileError, ErrorTargetExisting, ErrorFileLocked, X //accummulated delta != file size! consider ADS, sparse, compressed files - const IOCallback& notifyUnbufferedIO /*throw X*/); + const IOCallback& notifyUnbufferedIO /*throw X*/); } #endif //FILE_ACCESS_H_8017341345614857 diff --git a/zen/file_error.h b/zen/file_error.h index 101d6543..101d6543 100755..100644 --- a/zen/file_error.h +++ b/zen/file_error.h diff --git a/zen/file_id_def.h b/zen/file_id_def.h index 55ee77f5..55ee77f5 100755..100644 --- a/zen/file_id_def.h +++ b/zen/file_id_def.h diff --git a/zen/file_io.cpp b/zen/file_io.cpp index 1c6ab6f2..1c6ab6f2 100755..100644 --- a/zen/file_io.cpp +++ b/zen/file_io.cpp diff --git a/zen/file_io.h b/zen/file_io.h index bf23d22c..bf23d22c 100755..100644 --- a/zen/file_io.h +++ b/zen/file_io.h diff --git a/zen/file_traverser.cpp b/zen/file_traverser.cpp index cc6e0c0b..cc6e0c0b 100755..100644 --- a/zen/file_traverser.cpp +++ b/zen/file_traverser.cpp diff --git a/zen/file_traverser.h b/zen/file_traverser.h index 5c1683f8..5c1683f8 100755..100644 --- a/zen/file_traverser.h +++ b/zen/file_traverser.h diff --git a/zen/format_unit.cpp b/zen/format_unit.cpp index 3e75278b..3e75278b 100755..100644 --- a/zen/format_unit.cpp +++ b/zen/format_unit.cpp diff --git a/zen/format_unit.h b/zen/format_unit.h index de5a0811..de5a0811 100755..100644 --- a/zen/format_unit.h +++ b/zen/format_unit.h diff --git a/zen/globals.h b/zen/globals.h index 024147fa..024147fa 100755..100644 --- a/zen/globals.h +++ b/zen/globals.h diff --git a/zen/guid.h b/zen/guid.h index a26688f8..c89e8082 100755..100644 --- a/zen/guid.h +++ b/zen/guid.h @@ -18,13 +18,11 @@ namespace zen inline std::string generateGUID() //creates a 16-byte GUID { -#if __GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 25) //getentropy() requires glibc 2.25 (ldd --version) PS: Centos 7 is on 2.17 std::string guid(16, '\0'); - if (::getentropy(&guid[0], 16) != 0) //"The maximum permitted value for the length argument is 256" +#if __GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 25) //getentropy() requires glibc 2.25 (ldd --version) PS: CentOS 7 is on 2.17 + if (::getentropy(&guid[0], guid.size()) != 0) //"The maximum permitted value for the length argument is 256" throw std::runtime_error(std::string(__FILE__) + "[" + numberTo<std::string>(__LINE__) + "] Failed to generate GUID." + "\n" + utfTo<std::string>(formatSystemError(L"getentropy", errno))); - return guid; - #else class RandomGeneratorPosix { @@ -55,10 +53,9 @@ std::string generateGUID() //creates a 16-byte GUID const int fd_ = ::open("/dev/urandom", O_RDONLY | O_CLOEXEC); }; thread_local RandomGeneratorPosix gen; - std::string guid(16, '\0'); - gen.getBytes(&guid[0], 16); - return guid; + gen.getBytes(&guid[0], guid.size()); #endif + return guid; } } diff --git a/zen/http.cpp b/zen/http.cpp index 1f89bf20..1f89bf20 100755..100644 --- a/zen/http.cpp +++ b/zen/http.cpp diff --git a/zen/http.h b/zen/http.h index 5d84be2c..5d84be2c 100755..100644 --- a/zen/http.h +++ b/zen/http.h diff --git a/zen/i18n.h b/zen/i18n.h index 2ecee45a..2ecee45a 100755..100644 --- a/zen/i18n.h +++ b/zen/i18n.h diff --git a/zen/json.h b/zen/json.h new file mode 100644 index 00000000..374a3f14 --- /dev/null +++ b/zen/json.h @@ -0,0 +1,541 @@ +// ***************************************************************************** +// * This file is part of the FreeFileSync project. It is distributed under * +// * GNU General Public License: https://www.gnu.org/licenses/gpl-3.0 * +// * Copyright (C) Zenju (zenju AT freefilesync DOT org) - All Rights Reserved * +// ***************************************************************************** + +#ifndef JSON_H_0187348321748321758934215734 +#define JSON_H_0187348321748321758934215734 + +#include <zen/string_tools.h> + + +namespace zen +{ +//https://tools.ietf.org/html/rfc8259 +struct JsonValue +{ + enum class Type + { + null, // + boolean, //primitive types + number, // + string, // + object, + array, + }; + + explicit JsonValue() {} + explicit JsonValue(Type t) : type(t) {} + explicit JsonValue(bool b) : type(Type::boolean), primVal(b ? "true" : "false") {} + explicit JsonValue(int64_t num) : type(Type::number), primVal(numberTo<std::string>(num)) {} + explicit JsonValue(double num) : type(Type::number), primVal(numberTo<std::string>(num)) {} + explicit JsonValue(const std::string& str) : type(Type::string), primVal(str) {} + + Type type = Type::null; + std::string primVal; //for primitive types + std::map<std::string, std::unique_ptr<JsonValue>> objectVal; //"[...] most implementations of JSON libraries do not accept duplicate keys [...]" => fine! + std::vector<std::unique_ptr<JsonValue>> arrayVal; +}; + + +std::string serializeJson(const JsonValue& jval, + const std::string& lineBreak = "\r\n", + const std::string& indent = " "); //noexcept + + +struct JsonParsingError +{ + JsonParsingError(size_t rowNo, size_t colNo) : row(rowNo), col(colNo) {} + const size_t row; //beginning with 0 + const size_t col; // +}; +JsonValue parseJson(const std::string& stream); //throw JsonParsingError + + + +//helper functions for JsonValue access: +inline +const JsonValue* getChildFromJsonObject(const JsonValue& jvalue, const std::string& name) +{ + if (jvalue.type != JsonValue::Type::object) + return nullptr; + + auto it = jvalue.objectVal.find(name); + if (it == jvalue.objectVal.end()) + return nullptr; + + return it->second.get(); +} + + +inline +std::optional<std::string> getPrimitiveFromJsonObject(const JsonValue& jvalue, const std::string& name) +{ + if (const JsonValue* childValue = getChildFromJsonObject(jvalue, name)) + if (childValue->type != JsonValue::Type::object && + childValue->type != JsonValue::Type::array) + return childValue->primVal; + return std::nullopt; +} + + + + + +//---------------------- implementation ---------------------- +namespace json_impl +{ +namespace +{ +std::string jsonEscape(const std::string& str) +{ + std::string output; + for (const char c : str) + { + if (c == '"') output += "\\\""; //escaping mandatory + else if (c == '\\') output += "\\\\"; // + + else if (c == '\b') output += "\\b"; // + else if (c == '\f') output += "\\f"; // + else if (c == '\n') output += "\\n"; //prefer compact escaping + else if (c == '\r') output += "\\r"; // + else if (c == '\t') output += "\\t"; // + + else if (static_cast<unsigned char>(c) < 32) + { + const auto hexDigits = hexify(c); + output += "\\u00"; + output += hexDigits.first; + output += hexDigits.second; + } + else + output += c; + } + return output; +} + + +std::string jsonUnescape(const std::string& str) +{ + std::string output; + std::basic_string<impl::Char16> utf16Buf; + + auto flushUtf16 = [&] + { + if (!utf16Buf.empty()) + { + impl::UtfDecoder<impl::Char16> decoder(utf16Buf.c_str(), utf16Buf.size()); + while (std::optional<impl::CodePoint> cp = decoder.getNext()) + impl::codePointToUtf<char>(*cp, [&](char c) { output += c; }); + utf16Buf.clear(); + } + }; + auto writeOut = [&](char c) + { + flushUtf16(); + output += c; + }; + + for (auto it = str.begin(); it != str.end(); ++it) + { + const char c = *it; + + if (c == '\\') + { + ++it; + if (it == str.end()) //unexpected end! + { + writeOut(c); + break; + } + + const char c2 = *it; + if (c2 == '"' || + c2 == '\\' || + c2 == '/') + writeOut(c2); + else if (c2 == 'b') writeOut('\b'); + else if (c2 == 'f') writeOut('\f'); + else if (c2 == 'n') writeOut('\n'); + else if (c2 == 'r') writeOut('\r'); + else if (c2 == 't') writeOut('\t'); + + else if (c2 == 'u' && + str.end() - it >= 5 && + isHexDigit(it[1]) && + isHexDigit(it[2]) && + isHexDigit(it[3]) && + isHexDigit(it[4])) + { + utf16Buf += static_cast<impl::Char16>(static_cast<unsigned char>(unhexify(it[1], it[2])) * 256 + + static_cast<unsigned char>(unhexify(it[3], it[4]))); + it += 4; + } + else //unknown escape sequence! + { + writeOut(c); + writeOut(c2); + } + } + else + writeOut(c); + } + flushUtf16(); + return output; +} + + +void serialize(const JsonValue& jval, std::string& stream, + const std::string& lineBreak, + const std::string& indent, + size_t indentLevel) +{ + //unlike our XML serialization the caller is repsonsible for line breaks and indentation of *first* line + auto writeIndent = [&](size_t level) + { + for (size_t i = 0; i < level; ++i) + stream += indent; + }; + + switch (jval.type) + { + case JsonValue::Type::null: + stream += "null"; + break; + + case JsonValue::Type::boolean: + case JsonValue::Type::number: + stream += jval.primVal; + break; + + case JsonValue::Type::string: + stream += '"' + jsonEscape(jval.primVal) + '"'; + break; + + case JsonValue::Type::object: + stream += '{'; + if (!jval.objectVal.empty()) + { + for (auto it = jval.objectVal.begin(); it != jval.objectVal.end(); ++it) + { + const auto& [childName, childValue] = *it; + + if (it != jval.objectVal.begin()) + stream += ','; + + stream += lineBreak; + writeIndent(indentLevel + 1); + + stream += '"' + jsonEscape(childName) + "\":"; + + if ((childValue->type == JsonValue::Type::object && !childValue->objectVal.empty()) || + (childValue->type == JsonValue::Type::array && !childValue->arrayVal .empty())) + { + stream += lineBreak; + writeIndent(indentLevel + 1); + } + else if (!indent.empty()) + stream += ' '; + + serialize(*childValue, stream, lineBreak, indent, indentLevel + 1); + } + stream += lineBreak; + writeIndent(indentLevel); + } + stream += '}'; + break; + + case JsonValue::Type::array: + stream += '['; + if (!jval.arrayVal.empty()) + { + for (auto it = jval.arrayVal.begin(); it != jval.arrayVal.end(); ++it) + { + const auto& childValue = **it; + + if (it != jval.arrayVal.begin()) + stream += ','; + + stream += lineBreak; + writeIndent(indentLevel + 1); + + serialize(childValue, stream, lineBreak, indent, indentLevel + 1); + } + stream += lineBreak; + writeIndent(indentLevel); + } + stream += ']'; + break; + } +} +} +} + + +inline +std::string serializeJson(const JsonValue& jval, + const std::string& lineBreak, + const std::string& indent) //noexcept +{ + std::string output; + json_impl::serialize(jval, output, lineBreak, indent, 0); + output += lineBreak; + return output; +} + + +namespace json_impl +{ +struct Token +{ + enum class Type + { + eof, + curlyOpen, + curlyClose, + squareOpen, + squareClose, + colon, + comma, + string, // + number, //primitive types + boolean, // + null, // + }; + + Token(Type t) : type(t) {} + + Type type; + std::string primVal; //for primitive types +}; + +class Scanner +{ +public: + Scanner(const std::string& stream) : stream_(stream), pos_(stream_.begin()) + { + if (zen::startsWith(stream_, BYTE_ORDER_MARK_UTF8)) + pos_ += strLength(BYTE_ORDER_MARK_UTF8); + } + + Token getNextToken() //throw JsonParsingError + { + //skip whitespace + pos_ = std::find_if(pos_, stream_.end(), std::not_fn(isJsonWhiteSpace)); + + if (pos_ == stream_.end()) + return Token::Type::eof; + + if (*pos_ == '{') return ++pos_, Token::Type::curlyOpen; + if (*pos_ == '}') return ++pos_, Token::Type::curlyClose; + if (*pos_ == '[') return ++pos_, Token::Type::squareOpen; + if (*pos_ == ']') return ++pos_, Token::Type::squareClose; + if (*pos_ == ':') return ++pos_, Token::Type::colon; + if (*pos_ == ',') return ++pos_, Token::Type::comma; + if (startsWith("null")) return pos_ += 4, Token(Token::Type::null); + + if (startsWith("true")) + { + pos_ += 4; + Token tk(Token::Type::boolean); + tk.primVal = "true"; + return tk; + } + if (startsWith("false")) + { + pos_ += 5; + Token tk(Token::Type::boolean); + tk.primVal = "false"; + return tk; + } + + if (*pos_ == '"') + { + for (auto it = ++pos_; it != stream_.end(); ++it) + if (*it == '"') + { + Token tk(Token::Type::string); + tk.primVal = jsonUnescape({ pos_, it }); + pos_ = ++it; + return tk; + } + else if (*it == '\\') //skip next char + if (++it == stream_.end()) + break; + + throw JsonParsingError(posRow(), posCol()); + } + + //expect a number: + const auto itNumEnd = std::find_if(pos_, stream_.end(), std::not_fn(isJsonNumDigit)); + if (itNumEnd == pos_) + throw JsonParsingError(posRow(), posCol()); + + Token tk(Token::Type::number); + tk.primVal.assign(pos_, itNumEnd); + pos_ = itNumEnd; + return tk; + } + + size_t posRow() const //current row beginning with 0 + { + const size_t crSum = std::count(stream_.begin(), pos_, '\r'); //carriage returns + const size_t nlSum = std::count(stream_.begin(), pos_, '\n'); //new lines + assert(crSum == 0 || nlSum == 0 || crSum == nlSum); + return std::max(crSum, nlSum); //be compatible with Linux/Mac/Win + } + + size_t posCol() const //current col beginning with 0 + { + //seek beginning of line + for (auto it = pos_; it != stream_.begin(); ) + { + --it; + if (*it == '\r' || *it == '\n') + return pos_ - it - 1; + } + return pos_ - stream_.begin(); + } + +private: + Scanner (const Scanner&) = delete; + Scanner& operator=(const Scanner&) = delete; + + static bool isJsonWhiteSpace(char c) { return c == ' ' || c == '\t' || c == '\r' || c == '\n'; } + static bool isJsonNumDigit (char c) { return ('0' <= c && c <= '9') || c == '-' || c == '+' || c == '.' || c == 'e'|| c == 'E'; } + + bool startsWith(const std::string& prefix) const + { + return zen::startsWith(StringRef<const char>(pos_, stream_.end()), prefix); + } + + const std::string stream_; + std::string::const_iterator pos_; +}; + + +class JsonParser +{ +public: + JsonParser(const std::string& stream) : + scn_(stream), + tk_(scn_.getNextToken()) {} //throw JsonParsingError + + JsonValue parse() //throw JsonParsingError + { + JsonValue jval = parseValue(); //throw JsonParsingError + expectToken(Token::Type::eof); // + return jval; + } + +private: + JsonParser (const JsonParser&) = delete; + JsonParser& operator=(const JsonParser&) = delete; + + JsonValue parseValue() //throw JsonParsingError + { + if (token().type == Token::Type::curlyOpen) + { + nextToken(); //throw JsonParsingError + + JsonValue jval(JsonValue::Type::object); + + if (token().type != Token::Type::curlyClose) + for (;;) + { + expectToken(Token::Type::string); //throw JsonParsingError + std::string name = token().primVal; + nextToken(); //throw JsonParsingError + + consumeToken(Token::Type::colon); //throw JsonParsingError + + JsonValue value = parseValue(); //throw JsonParsingError + jval.objectVal.emplace(std::move(name), std::make_unique<JsonValue>(std::move(value))); + + if (token().type != Token::Type::comma) + break; + nextToken(); //throw JsonParsingError + } + + consumeToken(Token::Type::curlyClose); //throw JsonParsingError + return jval; + } + else if (token().type == Token::Type::squareOpen) + { + nextToken(); //throw JsonParsingError + + JsonValue jval(JsonValue::Type::array); + + if (token().type != Token::Type::squareClose) + for (;;) + { + JsonValue value = parseValue(); //throw JsonParsingError + jval.arrayVal.emplace_back(std::make_unique<JsonValue>(std::move(value))); + + if (token().type != Token::Type::comma) + break; + nextToken(); //throw JsonParsingError + } + + consumeToken(Token::Type::squareClose); //throw JsonParsingError + return jval; + } + else if (token().type == Token::Type::string) + { + JsonValue jval(token().primVal); + nextToken(); //throw JsonParsingError + return jval; + } + else if (token().type == Token::Type::number) + { + JsonValue jval(JsonValue::Type::number); + jval.primVal = token().primVal; + nextToken(); //throw JsonParsingError + return jval; + } + else if (token().type == Token::Type::boolean) + { + JsonValue jval(JsonValue::Type::boolean); + jval.primVal = token().primVal; + nextToken(); //throw JsonParsingError + return jval; + } + else if (token().type == Token::Type::null) + { + nextToken(); //throw JsonParsingError + return JsonValue(); + } + else //unexpected token + throw JsonParsingError(scn_.posRow(), scn_.posCol()); + } + + const Token& token() const { return tk_; } + + void nextToken() { tk_ = scn_.getNextToken(); } //throw JsonParsingError + + void expectToken(Token::Type t) //throw JsonParsingError + { + if (token().type != t) + throw JsonParsingError(scn_.posRow(), scn_.posCol()); + } + + void consumeToken(Token::Type t) //throw JsonParsingError + { + expectToken(t); //throw JsonParsingError + nextToken(); // + } + + Scanner scn_; + Token tk_; +}; +} + +inline +JsonValue parseJson(const std::string& stream) //throw JsonParsingError +{ + return json_impl::JsonParser(stream).parse(); //throw JsonParsingError +} +} + +#endif //JSON_H_0187348321748321758934215734 diff --git a/zen/legacy_compiler.h b/zen/legacy_compiler.h index 54605945..54605945 100755..100644 --- a/zen/legacy_compiler.h +++ b/zen/legacy_compiler.h diff --git a/zen/perf.h b/zen/perf.h index 77251f8c..77251f8c 100755..100644 --- a/zen/perf.h +++ b/zen/perf.h diff --git a/zen/process_priority.cpp b/zen/process_priority.cpp index e925f142..e925f142 100755..100644 --- a/zen/process_priority.cpp +++ b/zen/process_priority.cpp diff --git a/zen/process_priority.h b/zen/process_priority.h index cfadfff1..cfadfff1 100755..100644 --- a/zen/process_priority.h +++ b/zen/process_priority.h diff --git a/zen/recycler.cpp b/zen/recycler.cpp index 8d34f262..dc156a6f 100755..100644 --- a/zen/recycler.cpp +++ b/zen/recycler.cpp @@ -45,7 +45,7 @@ bool zen::recycleOrDeleteIfExists(const Zstring& itemPath) //throw FileError return true; } - throw FileError(errorMsg, formatSystemError(L"g_file_trash", L"Glib Error Code " + numberTo<std::wstring>(error->code), utfTo<std::wstring>(error->message))); + throw FileError(errorMsg, formatSystemError(L"g_file_trash", replaceCpy(_("Error Code %x"), L"%x", numberTo<std::wstring>(error->code)), utfTo<std::wstring>(error->message))); //g_quark_to_string(error->domain) } return true; diff --git a/zen/recycler.h b/zen/recycler.h index 0e5ebbb1..0e5ebbb1 100755..100644 --- a/zen/recycler.h +++ b/zen/recycler.h diff --git a/zen/ring_buffer.h b/zen/ring_buffer.h index e3dbd55f..e3dbd55f 100755..100644 --- a/zen/ring_buffer.h +++ b/zen/ring_buffer.h diff --git a/zen/scope_guard.h b/zen/scope_guard.h index 9eff6c1f..9eff6c1f 100755..100644 --- a/zen/scope_guard.h +++ b/zen/scope_guard.h diff --git a/zen/serialize.h b/zen/serialize.h index 8b4c58ea..5a38303c 100755..100644 --- a/zen/serialize.h +++ b/zen/serialize.h @@ -114,6 +114,7 @@ private: const IOCallback& notifyUnbufferedIO_; }; + //buffered input/output stream reference implementations: template <class BinContainer> struct MemoryStreamIn diff --git a/zen/shell_execute.h b/zen/shell_execute.h index 98824d70..98824d70 100755..100644 --- a/zen/shell_execute.h +++ b/zen/shell_execute.h diff --git a/zen/shutdown.cpp b/zen/shutdown.cpp index a25e9c64..f09c4d07 100755..100644 --- a/zen/shutdown.cpp +++ b/zen/shutdown.cpp @@ -40,30 +40,10 @@ void zen::terminateProcess(int exitCode) } -/* -Command line alternatives: - -#ifdef ZEN_WIN -#ifdef ZEN_WIN_VISTA_AND_LATER - Shut down: shutdown /s /t 60 - Sleep: rundll32.exe powrprof.dll,SetSuspendState Sleep - Log off: shutdown /l -#else //XP - Shut down: shutdown -s -t 60 - Standby: rundll32.exe powrprof.dll,SetSuspendState //this triggers standby OR hibernate, depending on whether hibernate setting is active! no suspend on XP? - Log off: shutdown -l -#endif - -#elif defined ZEN_LINUX - Shut down: systemctl poweroff //alternative requiring admin: sudo shutdown -h 1 - Sleep: systemctl suspend //alternative requiring admin: sudo pm-suspend - Log off: gnome-session-quit --no-prompt - //alternative requiring admin: sudo killall Xorg - //alternative without admin: dbus-send --session --print-reply --dest=org.gnome.SessionManager /org/gnome/SessionManager org.gnome.SessionManager.Logout uint32:1 - -#elif defined ZEN_MAC - Shut down: osascript -e 'tell application "System Events" to shut down' - Sleep: osascript -e 'tell application "System Events" to sleep' - Log off: osascript -e 'tell application "System Events" to log out' -#endif -*/ +//Command line alternatives: + //Shut down: systemctl poweroff //alternative requiring admin: sudo shutdown -h 1 + //Sleep: systemctl suspend //alternative requiring admin: sudo pm-suspend + //Log off: gnome-session-quit --no-prompt + // alternative requiring admin: sudo killall Xorg + // alternative without admin: dbus-send --session --print-reply --dest=org.gnome.SessionManager /org/gnome/SessionManager org.gnome.SessionManager.Logout uint32:1 + diff --git a/zen/shutdown.h b/zen/shutdown.h index 2d66d1e8..2d66d1e8 100755..100644 --- a/zen/shutdown.h +++ b/zen/shutdown.h diff --git a/zen/socket.h b/zen/socket.h index 33ac2e50..c77a4084 100755..100644 --- a/zen/socket.h +++ b/zen/socket.h @@ -119,7 +119,7 @@ size_t tryWriteSocket(SocketType socket, const void* buffer, size_t bytesToWrite int bytesWritten = 0; for (;;) { - bytesWritten = ::send(socket, //_In_ SOCKET s, + bytesWritten = ::send(socket, //_In_ SOCKET s, static_cast<const char*>(buffer), //_In_ const char *buf, static_cast<int>(bytesToWrite), //_In_ int len, 0); //_In_ int flags diff --git a/zen/stl_tools.h b/zen/stl_tools.h index f1ab7c16..f1ab7c16 100755..100644 --- a/zen/stl_tools.h +++ b/zen/stl_tools.h diff --git a/zen/string_base.h b/zen/string_base.h index 2247f93a..2247f93a 100755..100644 --- a/zen/string_base.h +++ b/zen/string_base.h diff --git a/zen/string_tools.h b/zen/string_tools.h index c3970d05..c3970d05 100755..100644 --- a/zen/string_tools.h +++ b/zen/string_tools.h diff --git a/zen/string_traits.h b/zen/string_traits.h index 93cfd81c..93cfd81c 100755..100644 --- a/zen/string_traits.h +++ b/zen/string_traits.h diff --git a/zen/symlink_target.h b/zen/symlink_target.h index 2393013e..2393013e 100755..100644 --- a/zen/symlink_target.h +++ b/zen/symlink_target.h diff --git a/zen/sys_error.cpp b/zen/sys_error.cpp index 2acaca1d..2acaca1d 100755..100644 --- a/zen/sys_error.cpp +++ b/zen/sys_error.cpp diff --git a/zen/sys_error.h b/zen/sys_error.h index a087172f..a087172f 100755..100644 --- a/zen/sys_error.h +++ b/zen/sys_error.h diff --git a/zen/thread.cpp b/zen/thread.cpp index 08bfaa25..08bfaa25 100755..100644 --- a/zen/thread.cpp +++ b/zen/thread.cpp diff --git a/zen/thread.h b/zen/thread.h index 5e298ba1..52b85d7f 100755..100644 --- a/zen/thread.h +++ b/zen/thread.h @@ -186,7 +186,7 @@ public: std::future<void> allDone = promiseDone->get_future(); notifyWhenDone([promiseDone] { promiseDone->set_value(); }); //std::function doesn't support construction involving move-only types! - //use reference? => not guaranteed safe, e.g. promise object theoretically might be accessed inside set_value() after future gets signalled + //use reference? => potential lifetime issue, e.g. promise object theoretically might be accessed inside set_value() after future gets signalled allDone.get(); } diff --git a/zen/time.h b/zen/time.h index a32e28e3..a32e28e3 100755..100644 --- a/zen/time.h +++ b/zen/time.h diff --git a/zen/type_traits.h b/zen/type_traits.h index 8783cb6a..8783cb6a 100755..100644 --- a/zen/type_traits.h +++ b/zen/type_traits.h diff --git a/zen/utf.h b/zen/utf.h index 5a095874..5a095874 100755..100644 --- a/zen/utf.h +++ b/zen/utf.h diff --git a/zen/warn_static.h b/zen/warn_static.h index fb8fbb95..fb8fbb95 100755..100644 --- a/zen/warn_static.h +++ b/zen/warn_static.h diff --git a/zen/zlib_wrap.cpp b/zen/zlib_wrap.cpp index fbbe2f09..9eb4302f 100755..100644 --- a/zen/zlib_wrap.cpp +++ b/zen/zlib_wrap.cpp @@ -6,9 +6,9 @@ #include "zlib_wrap.h" //Windows: use the SAME zlib version that wxWidgets is linking against! //C:\Data\Projects\wxWidgets\Source\src\zlib\zlib.h -//Linux/macOS: use zlib system header for both wxWidgets and Curl (zlib is required for HTTP) +//Linux/macOS: use zlib system header for both wxWidgets and libcurl (zlib is required for HTTP) // => don't compile wxWidgets with: --with-zlib=builtin -#include <zlib.h> +#include <zlib.h> //https://www.zlib.net/manual.html using namespace zen; @@ -51,3 +51,78 @@ size_t zen::impl::zlib_decompress(const void* src, size_t srcLen, void* trg, siz throw ZlibInternalError(); return bufferSize; } + + +class InputStreamAsGzip::Impl +{ +public: + Impl(const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/) : //throw ZlibInternalError; returning 0 signals EOF: Posix read() semantics + readBlock_(readBlock) + { + const int windowBits = MAX_WBITS + 16; //"add 16 to windowBits to write a simple gzip header" + + //"memLevel=1 uses minimum memory but is slow and reduces compression ratio; memLevel=9 uses maximum memory for optimal speed. + const int memLevel = 9; //test; 280 MB installer file: level 9 shrinks runtime by ~8% compared to level 8 (==DEF_MEM_LEVEL) at the cost of 128 KB extra memory + static_assert(memLevel <= MAX_MEM_LEVEL); + + const int rv = ::deflateInit2(&gzipStream_, //z_streamp strm + 3 /*see db_file.cpp*/, //int level + Z_DEFLATED, //int method + windowBits, //int windowBits + memLevel, //int memLevel + Z_DEFAULT_STRATEGY); //int strategy + if (rv != Z_OK) + throw ZlibInternalError(); + } + + ~Impl() + { + const int rv = ::deflateEnd(&gzipStream_); + assert(rv == Z_OK); + (void)rv; + } + + size_t read(void* buffer, size_t bytesToRead) //throw ZlibInternalError, X; return "bytesToRead" bytes unless end of stream! + { + if (bytesToRead == 0) //"read() with a count of 0 returns zero" => indistinguishable from end of file! => check! + throw std::logic_error("Contract violation! " + std::string(__FILE__) + ":" + numberTo<std::string>(__LINE__)); + + gzipStream_.next_out = static_cast<Bytef*>(buffer); + gzipStream_.avail_out = static_cast<uInt>(bytesToRead); + + for (;;) + { + if (gzipStream_.avail_in == 0 && !eof_) + { + if (bufIn_.size() < bytesToRead) + bufIn_.resize(bytesToRead); + + const size_t bytesRead = readBlock_(&bufIn_[0], bufIn_.size()); //throw X; returning 0 signals EOF: Posix read() semantics + gzipStream_.next_in = reinterpret_cast<z_const Bytef*>(&bufIn_[0]); + gzipStream_.avail_in = static_cast<uInt>(bytesRead); + if (bytesRead == 0) + eof_ = true; + } + + const int rv = ::deflate(&gzipStream_, eof_ ? Z_FINISH : Z_NO_FLUSH); + if (rv == Z_STREAM_END) + return bytesToRead - gzipStream_.avail_out; + if (rv != Z_OK) + throw ZlibInternalError(); + + if (gzipStream_.avail_out == 0) + return bytesToRead; + } + } + +private: + const std::function<size_t(void* buffer, size_t bytesToRead)> readBlock_; //throw X + bool eof_ = false; + std::vector<std::byte> bufIn_; + z_stream gzipStream_ = {}; +}; + + +zen::InputStreamAsGzip::InputStreamAsGzip(const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/) : pimpl_(std::make_unique<Impl>(readBlock)) {} //throw ZlibInternalError +zen::InputStreamAsGzip::~InputStreamAsGzip() {} +size_t zen::InputStreamAsGzip::read(void* buffer, size_t bytesToRead) { return pimpl_->read(buffer, bytesToRead); } //throw ZlibInternalError, X diff --git a/zen/zlib_wrap.h b/zen/zlib_wrap.h index b92a8eba..c8647baf 100755..100644 --- a/zen/zlib_wrap.h +++ b/zen/zlib_wrap.h @@ -25,8 +25,19 @@ template <class BinContainer> BinContainer decompress(const BinContainer& stream); //throw ZlibInternalError +class InputStreamAsGzip //convert input stream into gzip on the fly +{ +public: + InputStreamAsGzip( //throw ZlibInternalError + const std::function<size_t(void* buffer, size_t bytesToRead)>& readBlock /*throw X*/); //returning 0 signals EOF: Posix read() semantics + ~InputStreamAsGzip(); + size_t read(void* buffer, size_t bytesToRead); //throw ZlibInternalError, X; return "bytesToRead" bytes unless end of stream! +private: + class Impl; + const std::unique_ptr<Impl> pimpl_; +}; @@ -52,9 +63,7 @@ BinContainer compress(const BinContainer& stream, int level) //throw ZlibInterna //save uncompressed stream size for decompression const uint64_t uncompressedSize = stream.size(); //use portable number type! contOut.resize(sizeof(uncompressedSize)); - std::copy(reinterpret_cast<const std::byte*>(&uncompressedSize), - reinterpret_cast<const std::byte*>(&uncompressedSize) + sizeof(uncompressedSize), - &*contOut.begin()); + std::memcpy(&*contOut.begin(), &uncompressedSize, sizeof(uncompressedSize)); const size_t bufferEstimate = impl::zlib_compressBound(stream.size()); //upper limit for buffer size, larger than input size!!! @@ -83,9 +92,9 @@ BinContainer decompress(const BinContainer& stream) //throw ZlibInternalError uint64_t uncompressedSize = 0; //use portable number type! if (stream.size() < sizeof(uncompressedSize)) throw ZlibInternalError(); - std::copy(&*stream.begin(), - &*stream.begin() + sizeof(uncompressedSize), - reinterpret_cast<std::byte*>(&uncompressedSize)); + + std::memcpy(&uncompressedSize, &*stream.begin(), sizeof(uncompressedSize)); + //attention: contOut MUST NOT be empty! Else it will pass a nullptr to zlib_decompress() => Z_STREAM_ERROR although "uncompressedSize == 0"!!! //secondary bug: don't dereference iterator into empty container! if (uncompressedSize == 0) //cannot be 0: compress() directly maps empty -> empty container skipping zlib! diff --git a/zen/zstring.cpp b/zen/zstring.cpp index f8a34045..62a0caef 100755..100644 --- a/zen/zstring.cpp +++ b/zen/zstring.cpp @@ -57,7 +57,7 @@ Zstring getUnicodeNormalForm(const Zstring& str) // const char* precomposed = "\xc3\xb3"; try { - gchar* outStr = ::g_utf8_normalize (str.c_str(), str.length(), G_NORMALIZE_DEFAULT_COMPOSE); + gchar* outStr = ::g_utf8_normalize(str.c_str(), str.length(), G_NORMALIZE_DEFAULT_COMPOSE); if (!outStr) throw SysError(L"g_utf8_normalize: conversion failed. (" + utfTo<std::wstring>(str) + L")"); ZEN_ON_SCOPE_EXIT(::g_free(outStr)); diff --git a/zen/zstring.h b/zen/zstring.h index 6727253b..6727253b 100755..100644 --- a/zen/zstring.h +++ b/zen/zstring.h diff --git a/zenXml/zenxml/cvrt_struc.h b/zenXml/zenxml/cvrt_struc.h index 85c5d8d0..85c5d8d0 100755..100644 --- a/zenXml/zenxml/cvrt_struc.h +++ b/zenXml/zenxml/cvrt_struc.h diff --git a/zenXml/zenxml/cvrt_text.h b/zenXml/zenxml/cvrt_text.h index 51b23173..51b23173 100755..100644 --- a/zenXml/zenxml/cvrt_text.h +++ b/zenXml/zenxml/cvrt_text.h diff --git a/zenXml/zenxml/dom.h b/zenXml/zenxml/dom.h index d95f5aa1..d95f5aa1 100755..100644 --- a/zenXml/zenxml/dom.h +++ b/zenXml/zenxml/dom.h diff --git a/zenXml/zenxml/parser.h b/zenXml/zenxml/parser.h index 70bf6654..70bf6654 100755..100644 --- a/zenXml/zenxml/parser.h +++ b/zenXml/zenxml/parser.h diff --git a/zenXml/zenxml/xml.h b/zenXml/zenxml/xml.h index 27701248..27701248 100755..100644 --- a/zenXml/zenxml/xml.h +++ b/zenXml/zenxml/xml.h |