Dev to Master - 5.4.0-preview (#853)

* Fixed the potential error reported by Prefast code analysis

* Use SQLSRV_ASSERT for checking NULL ptrs

* For these AKV tests check env despite not AE connected

* Added the driver option to run functional tests

* Fixed connection pooling tests for more than one ODBC drivers

* added driver option to pdo isPooled.php

* Removed win32 ifdefs re connection resiliency (#802)

* Set the driver argument for getDSN to null by default (#798)

* Added the driver argument to getDSN

* Dropped the driver argument but set to null as default

* Removed the AE condition in locale support

* Modified the AE condition for locale support

* Changed int to SQLLEN to avoid infinite loop (#806)

* Version 5.3.0 (#803)

* Version 5.3.0

* Fixed the wrong replacements

* Added comments block to m4 files

* Use dnl for comments

*  Modified AE fetch phptypes test to insert only one row at a time and loop through php types (#801)

* Modified AE fetch phptypes test to insert only one row at a time and loop through php types

* Fixed formatting

* Streamlined two very similar large column name tests (#807)

* Streamlined two very similar large column name tests

* Changed the EOL

* Updates to change log and readme (#811)

* Updates to change log and readme

* Dropped support for Ubuntu 17

* Modified as per review comments

* Fixed connection resiliency tests for Unix, updated AppVeyor for ODBC 17.2

* Fixed expected output

* Fixed output and skipifs

* Fixed skipifs and output

* Fixed driver name

* Updated installation instructions and sample script (#813)

* Updated instructions and sample test for 5.3.0 RTW

* Fixed sample code to adhere to php coding standard

* Fixed cases and spaces

* Modified NOTE for UB 18.04 based on review comments

* Added 'exit'

* Modified change log and readme based on review to PR 811

* Applied review comments

* build output to debug appveyor failure

* removed debug output

* Streamlined two very similar large column name tests (#815)

* Streamlined two very similar large column name tests

* Added random number of test table names to avoid operand clash issues

* Replaced to with for based on review

* Changelog updated

* changelog updated, test skipif changed to run on unix platforms

* Fixed skipif typo

* Fixed typo in skipif for pdo

* Fixed some output for Travis

* Moved error checking inside pdo connres tests

* Added links back to changelog

* Fixed output for sqlsrv connres tests

* Fixed output

* Fixed output again

* Fixed skipifs for connres

* Tweaked per review comments

* Changes made to source and tests to support PHP 7.3 (#822)

* Changes made to support php 7.3

* Correct use of the smart pointer

* Fixed the tests for 7.3

* Some clean up for array_init()

* Fixed formattings and clean up

* One more fix

* Initialising strings with nulls

* Removed some spaces

* Made array index spacing consistent

* Fix for compilation problem

* Fix for compilation problem again

* Before freeing stmt in destructor check if dbh driver data is NULL  (#829)

* Issue 434 - set dbh driver data to NULL as well in destructor

* Reverted the last change but instead check if dbh driver_data is already freed

* Modified the comment

* Added driver to the skipif conditions (#831)

* Used git clone instead to download source from a branch of a tag (#832)

* Modified the error handling to make it more flexible (#833)

* Made error handling more flexible

* Fixed a minor issue with a test

* Enabled Spectre Mitigations (#836)

* Incorporated changes in PR 634 to pdo_sqlsrv (#834)

* Incorporated changes in PR 634 to pdo_sqlsrv

* Reverted the changes because the array is for internal use only

* Modified README re user's suggestion (#841)

* Modified README re user's suggestion

* Moved the if condition to the end as per review

* Adding supporting for Azure AD access token (#837)

* Adding supporting for Azure AD access token

* Added more comments for the AD access token skipif files

* Save the pointer to access token struct until after connecting

* Clear the access token data before freeing the memory

* Added a reference as per review

* Feature request - new PDO_STMT_OPTION_FETCHES_DATETIME_TYPE flag for pdo_sqlsrv to return datetime as objects (#842)

* Feature request - issue 648

* Fixed constructor for field_cache and added another test

* Added tests for FETCH_BOUND

* Added a new test for output param

* Modified output param test to set attributes differently

* Removed a useless helped function in a test

* Combined two new tests into one as per review

* Uncommented dropTable

* Feature request - add ReturnDatesAsStrings option to statement level for sqlsrv  (#844)

* Added ReturnDatesAsStrings option to the statement level

* Added new tests for ReturnDatesAsStrings at statement level

* Added more datetime types as per review

* Updated version 5.4.0-preview (#846)

* Updated version 5.4.0-preview

* Replaced 5.3 with 5.4

* Fixed sqlsrv datetime tests to connect with ColumnEncryption variables (#849)

* Change log for 5.4.0-preview (#850)

* Updated change log for 5.4.0-preview

* Updated 5.4.0 preview to add two new feature requests

* Modified change log as per review

* Modified the wordings

* Updated readme, changelog, and install instructions

CHANGELOG.md

@@ -3,6 +3,41 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/) 
 
+## 5.4.0-preview - 2018-09-24
+Updated PECL release packages. Here is the list of updates:
+
+### Added
+- Added support for PHP 7.3.0 RC 1
+- Added support for Azure AD Access Token (in Linux / macOS this requires [MS ODBC Driver 17+](https://docs.microsoft.com/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server) and [unixODBC](http://www.unixodbc.org/) 2.3.6+)
+- Feature Request [#842](https://github.com/Microsoft/msphpsql/pull/842) - new PDO_STMT_OPTION_FETCHES_DATETIME_TYPE flag for pdo_sqlsrv to return datetime as objects
+- Feature Request [#844](https://github.com/Microsoft/msphpsql/pull/844) - add ReturnDatesAsStrings option to statement level for sqlsrv
+- Compatible with [ODBC Driver 17.3 CTP](https://blogs.msdn.microsoft.com/sqlnativeclient/2018/09/24/odbc-driver-17-3-preview-for-sql-server-released/)
+
+### Removed
+- Dropped support for Ubuntu 17.10
+- Dropped support for PHP 7.0 - [Version 5.3](https://docs.microsoft.com/sql/connect/php/system-requirements-for-the-php-sql-driver?view=sql-server-2017) is the last to support PHP 7.0.
+
+### Fixed
+- Issue [#434](https://github.com/Microsoft/msphpsql/issues/434) - To avoid the pitfall that could result in a crash, before freeing stmt in the destructor check if its dbh driver data is NULL
+- Pull Request [#836](https://github.com/Microsoft/msphpsql/pull/836) - Modified the config files to enable Spectre Mitigations (use /Qspectre switch) for PHP 7.2
+- Pull Request [#833](https://github.com/Microsoft/msphpsql/pull/833) - Streamlined the error handling to remove a potential cause of crash
+
+### Limitations
+- No support for inout / output params when using sql_variant type
+- In Linux and macOS, setlocale() only takes effect if it is invoked before the first connection. Attempting to set the locale after connecting will not work
+- Always Encrypted requires [MS ODBC Driver 17+](https://docs.microsoft.com/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server)
+  - Only Windows Certificate Store and Azure Key Vault are supported. Custom Keystores are not yet supported
+  - Issue [#716](https://github.com/Microsoft/msphpsql/issues/716) - With Always Encrypted enabled, named parameters in subqueries are not supported
+  - [Always Encrypted limitations](https://docs.microsoft.com/sql/connect/php/using-always-encrypted-php-drivers#limitations-of-the-php-drivers-when-using-always-encrypted)
+
+### Known Issues
+- Connection pooling on Linux or macOS is not recommended with [unixODBC](http://www.unixodbc.org/) < 2.3.7
+- When pooling is enabled in Linux or macOS
+  - unixODBC <= 2.3.4 (Linux and macOS) might not return proper diagnostic information, such as error messages, warnings and informative messages
+  - due to this unixODBC bug, fetch large data (such as xml, binary) as streams as a workaround. See the examples [here](https://github.com/Microsoft/msphpsql/wiki/Features#pooling)
+- With ColumnEncryption enabled, calling stored procedures with XML parameters does not work (Issue [#674](https://github.com/Microsoft/msphpsql/issues/674))
+- With ColumnEncryption enabled, fetching varbinary(max), varchar(max) or nvarchar(max) may fail with [ODBC Driver 17.3 CTP](https://blogs.msdn.microsoft.com/sqlnativeclient/2018/09/24/odbc-driver-17-3-preview-for-sql-server-released/)
+
 ## 5.3.0 - 2018-07-20
 Updated PECL release packages. Here is the list of updates:
 

Linux-mac-install.md

@@ -5,13 +5,13 @@ These instructions install PHP 7.2 by default -- see the notes at the beginning
 
 ## Contents of this page:
 
-- [Installing the drivers on Ubuntu 16.04, 17.10, and 18.04](#installing-the-drivers-on-ubuntu-1604-1710-and-1804)
+- [Installing the drivers on Ubuntu 16.04 and 18.04](#installing-the-drivers-on-ubuntu-1604-and-1804)
 - [Installing the drivers on Red Hat 7](#installing-the-drivers-on-red-hat-7)
 - [Installing the drivers on Debian 8 and 9](#installing-the-drivers-on-debian-8-and-9)
 - [Installing the drivers on Suse 12](#installing-the-drivers-on-suse-12)
 - [Installing the drivers on macOS El Capitan, Sierra and High Sierra](#installing-the-drivers-on-macos-el-capitan-sierra-and-high-sierra)
 
-## Installing the drivers on Ubuntu 16.04, 17.10 and 18.04
+## Installing the drivers on Ubuntu 16.04 and 18.04
 
 > [!NOTE]
 > To install PHP 7.0 or 7.1, replace 7.2 with 7.0 or 7.1 in the following commands.

README.md

@@ -2,9 +2,9 @@
 
 **Welcome to the Microsoft Drivers for PHP for Microsoft SQL Server**
 
-The Microsoft Drivers for PHP for Microsoft SQL Server are PHP extensions that allow for the reading and writing of SQL Server data from within PHP scripts. The SQLSRV extension provides a procedural interface while the PDO_SQLSRV extension implements PHP Data Objects (PDO) for accessing data in all editions of SQL Server 2008 R2 and later (including Azure SQL DB). These drivers rely on the [Microsoft ODBC Driver for SQL Server](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017) to handle the low-level communication with SQL Server.
+The Microsoft Drivers for PHP for Microsoft SQL Server are PHP extensions that allow for the reading and writing of SQL Server data from within PHP scripts. The SQLSRV extension provides a procedural interface while the PDO_SQLSRV extension implements PHP Data Objects (PDO) for accessing data in all editions of SQL Server 2008 R2 and later (including Azure SQL DB). These drivers rely on the [Microsoft ODBC Driver for SQL Server](https://docs.microsoft.com/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017) to handle the low-level communication with SQL Server.
 
-This release contains the SQLSRV and PDO_SQLSRV drivers for PHP 7.* with improvements on both drivers and some limitations (see Limitations below for details). Upcoming releases will contain additional functionalities, bug fixes, and more.
+This release contains the SQLSRV and PDO_SQLSRV drivers for PHP 7.1+ with improvements on both drivers and some [limitations](https://github.com/Microsoft/msphpsql/releases). Upcoming releases will contain additional functionalities, bug fixes, and more.
 
 ## Take our survey
 
@@ -28,11 +28,11 @@ Thank you for taking the time to participate in our last survey. You can continu
 
 ## Get Started
 
-* [**Windows + SQL Server + PHP 7**](https://www.microsoft.com/en-us/sql-server/developer-get-started/php/windows)
-* [**Ubuntu + SQL Server + PHP 7**](https://www.microsoft.com/en-us/sql-server/developer-get-started/php/ubuntu)
-* [**RedHat + SQL Server + PHP 7**](https://www.microsoft.com/en-us/sql-server/developer-get-started/php/rhel)
-* [**SUSE + SQL Server + PHP 7**](https://www.microsoft.com/en-us/sql-server/developer-get-started/php/sles)
-* [**macOS + SQL Server + PHP 7**](https://www.microsoft.com/en-us/sql-server/developer-get-started/php/mac/)
+* [**Windows + SQL Server + PHP 7**](https://www.microsoft.com/sql-server/developer-get-started/php/windows)
+* [**Ubuntu + SQL Server + PHP 7**](https://www.microsoft.com/sql-server/developer-get-started/php/ubuntu)
+* [**RedHat + SQL Server + PHP 7**](https://www.microsoft.com/sql-server/developer-get-started/php/rhel)
+* [**SUSE + SQL Server + PHP 7**](https://www.microsoft.com/sql-server/developer-get-started/php/sles)
+* [**macOS + SQL Server + PHP 7**](https://www.microsoft.com/sql-server/developer-get-started/php/mac/)
 * [**Docker**](https://hub.docker.com/r/lbosqmsft/mssql-php-msphpsql/)
 
 
@@ -42,12 +42,12 @@ Thank you for taking the time to participate in our last survey. You can continu
 
 ## Prerequisites
 
-For full details on the system requirements for the drivers, see the [system requirements](https://docs.microsoft.com/en-us/sql/connect/php/system-requirements-for-the-php-sql-driver) on Microsoft Docs.
+For full details on the system requirements for the drivers, see the [system requirements](https://docs.microsoft.com/sql/connect/php/system-requirements-for-the-php-sql-driver) on Microsoft Docs.
 
 On the client machine:
-- PHP 7.0.x, 7.1.x, or 7.2.x (7.2.0 and up on Unix, 7.2.1 and up on Windows)
-- A Web server such as Internet Information Services (IIS) is required. Your Web server must be configured to run PHP
-- [Microsoft ODBC Driver 17, Microsoft ODBC Driver 13, or Microsoft ODBC Driver 11](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-2017)
+- PHP 7.1.x, or 7.2.x (7.2.0 and up on Unix, 7.2.1 and up on Windows)
+- [Microsoft ODBC Driver 17, Microsoft ODBC Driver 13, or Microsoft ODBC Driver 11](https://docs.microsoft.com/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-2017)
+- If using a Web server such as Internet Information Services (IIS) or Apache, it must be configured to run PHP
 
 On the server side, Microsoft SQL Server 2008 R2 and above on Windows are supported, as are Microsoft SQL Server 2016 and above on Linux.
 
@@ -57,16 +57,16 @@ The drivers are distributed as pre-compiled extensions for PHP found on the [rel
 
 If you choose to build the drivers, you must be able to build PHP 7 without including these extensions. For help building PHP on Windows, see the [official PHP website][phpbuild]. For details on compiling the drivers, see the [documentation](https://github.com/Microsoft/msphpsql/tree/dev/buildscripts#windows) -- an example buildscript is provided, but you can also compile the drivers manually.
 
-To load the drivers, make sure that the driver is in your PHP extension directory and enable it in your PHP installation's php.ini file by adding `extension=php_sqlsrv.dll` and/or `extension=php_pdo_sqlsrv.dll` to it.  If necessary, specify the extension directory using `extension_dir`, for example: `extension_dir = "C:\PHP\ext"`. Note that the precompiled binaries have different names -- substitute accordingly in php.ini. For more details on loading the drivers, see [Loading the PHP SQL Driver](https://docs.microsoft.com/en-us/sql/connect/php/loading-the-php-sql-driver) on Microsoft Docs.
+To load the drivers, make sure that the driver is in your PHP extension directory and enable it in your PHP installation's php.ini file by adding `extension=php_sqlsrv.dll` and/or `extension=php_pdo_sqlsrv.dll` to it.  If necessary, specify the extension directory using `extension_dir`, for example: `extension_dir = "C:\PHP\ext"`. Note that the precompiled binaries have different names -- substitute accordingly in php.ini. For more details on loading the drivers, see [Loading the PHP SQL Driver](https://docs.microsoft.com/sql/connect/php/loading-the-php-sql-driver) on Microsoft Docs.
 
-Finally, restart the Web server.
+Finally, if running PHP in a Web server, restart the Web server.
 
 ## Install (UNIX)
 
-For full instructions on installing the drivers on all supported Unix platforms, see [the installation instructions on Microsoft Docs](https://docs.microsoft.com/en-us/sql/connect/php/installation-tutorial-linux-mac).
+For full instructions on installing the drivers on all supported Unix platforms, see [the installation instructions on Microsoft Docs](https://docs.microsoft.com/sql/connect/php/installation-tutorial-linux-mac).
 
 ## Sample Code
-For PHP code samples, please see the [sample](https://github.com/Microsoft/msphpsql/tree/master/sample) folder or the [code samples on Microsoft Docs](https://docs.microsoft.com/en-us/sql/connect/php/code-samples-for-php-sql-driver).
+For PHP code samples, please see the [sample](https://github.com/Microsoft/msphpsql/tree/master/sample) folder or the [code samples on Microsoft Docs](https://docs.microsoft.com/sql/connect/php/code-samples-for-php-sql-driver).
 
 ## Limitations and Known Issues
 Please refer to [Releases](https://github.com/Microsoft/msphpsql/releases) for the latest limitations and known issues.
@@ -138,6 +138,6 @@ This project has adopted the Microsoft Open Source Code of Conduct. For more inf
 
 [phpbuild]: https://wiki.php.net/internals/windows/stepbystepbuild
 
-[phpdoc]: https://docs.microsoft.com/en-us/sql/connect/php/microsoft-php-driver-for-sql-server?view=sql-server-2017
+[phpdoc]: https://docs.microsoft.com/sql/connect/php/microsoft-php-driver-for-sql-server?view=sql-server-2017
 
 [PHPMan]: http://php.net/manual/install.unix.php

buildscripts/builddrivers.py

@@ -239,7 +239,7 @@ def validate_input(question, values):
     parser.add_argument('--DRIVER', default='all', choices=['all', 'sqlsrv', 'pdo_sqlsrv'], help="driver to build (default: all)")
     parser.add_argument('--DEBUG', action='store_true', help="enable debug mode (default: False)")
     parser.add_argument('--REPO', default='Microsoft', help="GitHub repository (default: Microsoft)")
-    parser.add_argument('--BRANCH', default='dev', help="GitHub repository branch (default: dev)")
+    parser.add_argument('--BRANCH', default='dev', help="GitHub repository branch or tag (default: dev)")
     parser.add_argument('--SOURCE', default=None, help="a local path to source file (default: None)")
     parser.add_argument('--TESTING', action='store_true', help="turns on testing mode (default: False)")
     parser.add_argument('--DESTPATH', default=None, help="an alternative destination for the drivers (default: None)")
@@ -280,7 +280,7 @@ def validate_input(question, values):
         answer = input("Download source from a GitHub repo? [y/n]: ")
         if answer == 'yes' or answer == 'y' or answer == '':
             repo = input("Name of the repo (hit enter for 'Microsoft'): ")
-            branch = input("Name of the branch (hit enter for 'dev'): ")
+            branch = input("Name of the branch or tag (hit enter for 'dev'): ")
             if repo == '':
                 repo = 'Microsoft'
             if branch == '':

buildscripts/buildtools.py

@@ -193,48 +193,33 @@ def write_lines_to_copy_source(driver, file):
         file.write('@CALL ROBOCOPY ' + source + ' ' + dest + ' /s /xx /xo' + os.linesep)
 
     @staticmethod
-    def download_msphpsql_source(repo, branch, dest_folder = 'Source', clean_up = True):
+    def download_msphpsql_source(repo, branch, dest_folder = 'Source'):
         """Download to *dest_folder* the msphpsql archive of the specified 
         GitHub *repo* and *branch*. The downloaded files will be removed by default.
         """
         try:
             work_dir = os.path.dirname(os.path.realpath(__file__))   
 
             temppath = os.path.join(work_dir, 'temp')
-            if os.path.exists(temppath):
-                shutil.rmtree(temppath)
-            os.makedirs(temppath)
+            # There is no need to remove tree - 
+            # for Bamboo, it will be cleaned up eventually
+            # for local development, this can act as a cached copy of the repo
+            if not os.path.exists(temppath):
+                os.makedirs(temppath)
             os.chdir(temppath)
 
-            file = branch + '.zip'
-            url = 'https://github.com/' + repo + '/msphpsql/archive/' + branch + '.zip'
-
-            print('Downloading ' + url + ' ...')
-            try:
-                with urllib.request.urlopen(url) as response, open(file, 'wb') as out_file:
-                    shutil.copyfileobj(response, out_file)
-            except:
-                print ("Resort to skip ssl verification...")
-                # need to skip ssl verification on some agents
-                # see https://www.python.org/dev/peps/pep-0476/
-                with urllib.request.urlopen(url, context=ssl._create_unverified_context()) as response, open(file, 'wb') as out_file:
-                    shutil.copyfileobj(response, out_file)
-
-            print('Extracting ' + file + ' ...')
-            zip = zipfile.ZipFile(file)
-            zip.extractall()
-            zip.close()
-
             msphpsqlFolder = os.path.join(temppath, 'msphpsql-' + branch)
+
+            url = 'https://github.com/' + repo + '/msphpsql.git'
+            command = 'git clone ' + url + ' -b ' + branch + ' --single-branch --depth 1 ' + msphpsqlFolder
+            os.system(command)
+
             source = os.path.join(msphpsqlFolder, 'source')
             os.chdir(work_dir)
 
             os.system('ROBOCOPY ' + source + '\shared ' + dest_folder + '\shared /xx /xo')
             os.system('ROBOCOPY ' + source + '\pdo_sqlsrv ' + dest_folder + '\pdo_sqlsrv /xx /xo')
             os.system('ROBOCOPY ' + source + '\sqlsrv ' + dest_folder + '\sqlsrv /xx /xo')
-
-            if clean_up:
-                shutil.rmtree(temppath)        
 
         except:
             print('Error occurred when downloading source')

source/pdo_sqlsrv/config.m4

@@ -4,7 +4,7 @@ dnl
 dnl  Contents: the code that will go into the configure script, indicating options, 
 dnl  external libraries and includes, and what source files are to be compiled.
 dnl 
-dnl  Microsoft Drivers 5.3 for PHP for SQL Server
+dnl  Microsoft Drivers 5.4 for PHP for SQL Server
 dnl  Copyright(c) Microsoft Corporation
 dnl  All rights reserved.
 dnl  MIT License

source/pdo_sqlsrv/config.w32

@@ -3,7 +3,7 @@
 //
 // Contents: JScript build configuration used by buildconf.bat
 //
-// Microsoft Drivers 5.3 for PHP for SQL Server
+// Microsoft Drivers 5.4 for PHP for SQL Server
 // Copyright(c) Microsoft Corporation
 // All rights reserved.
 // MIT License
@@ -36,6 +36,9 @@ if( PHP_PDO_SQLSRV != "no" ) {
             ADD_FLAG( "CFLAGS_PDO_SQLSRV", "/Zi" );
             if (PHP_DEBUG != "yes") ADD_FLAG( "CFLAGS_PDO_SQLSRV", "/guard:cf /O2" );
             ADD_FLAG( "CFLAGS_PDO_SQLSRV", "/D ZEND_WIN32_FORCE_INLINE" );
+            if (VCVERS >= 1913) {
+                ADD_FLAG("CFLAGS_PDO_SQLSRV", "/Qspectre");
+            } 
             ADD_EXTENSION_DEP('pdo_sqlsrv', 'pdo');
             EXTENSION("pdo_sqlsrv", pdo_sqlsrv_src_class, PHP_PDO_SQLSRV_SHARED, "/DZEND_ENABLE_STATIC_TSRMLS_CACHE=1");
     } else {