Skip to content

Commit

Permalink
Merge pull request #488 from heroku/blackfire-agent-package
Browse files Browse the repository at this point in the history
Blackfire: tests, agent v2, separate agent package, blackfireio/integration-heroku buildpack support
  • Loading branch information
dzuelke authored Jun 25, 2021
2 parents fcd42f3 + 79f4553 commit 004442d
Show file tree
Hide file tree
Showing 20 changed files with 224 additions and 93 deletions.
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# heroku-buildpack-php CHANGELOG

## v194 (2021-06-25)

### ADD

- blackfire/2.4.2 [David Zuelke]
- ext-blackfire/1.62.0 [David Zuelke]

### CHG

- ext-blackfire installs blackfire agent as separate dependency [David Zuelke]
- ext-blackfire will use blackfire agent from https://github.com/blackfireio/integration-heroku if present [David Zuelke]

### FIX

- ext-blackfire attempts to instrument during web dyno startup [David Zuelke]

## v193 (2021-06-07)

### ADD
Expand Down
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ gem 'rspec-expectations'
gem 'sem_version'
gem "parallel_tests"
gem "rake"
gem "ansi"
6 changes: 4 additions & 2 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@ GIT
GEM
remote: https://rubygems.org/
specs:
ansi (1.5.0)
diff-lcs (1.4.4)
erubis (2.7.0)
excon (0.79.0)
excon (0.82.0)
heroics (0.1.2)
erubis (~> 2.0)
excon
Expand All @@ -25,7 +26,7 @@ GEM
moneta (1.0.0)
multi_json (1.15.0)
parallel (1.20.1)
parallel_tests (3.5.0)
parallel_tests (3.7.0)
parallel
platform-api (3.3.0)
heroics (~> 0.1.1)
Expand All @@ -51,6 +52,7 @@ PLATFORMS
ruby

DEPENDENCIES
ansi
heroku_hatchet!
parallel_tests
rake
Expand Down
13 changes: 13 additions & 0 deletions bin/util/platform.php
Original file line number Diff line number Diff line change
Expand Up @@ -169,11 +169,24 @@ function mkmetas($package, array &$metapaks, &$have_runtime_req = false) {

preg_match("#^([^-]+)(?:-([0-9]+))?\$#", $STACK, $stack);
$provide = ["heroku-sys/".$stack[1] => (isset($stack[2])?$stack[2]:"1").gmdate(".Y.m.d")]; # heroku: 20.2021.02.04 etc

$replace = [];
// check whether the blackfire CLI is already there (from their https://github.com/blackfireio/integration-heroku buildpack)
exec("blackfire --no-ansi version 2>/dev/null", $blackfire_version, $have_blackfire);
if($have_blackfire === 0 && preg_match("/^Blackfire version (\d+\.\d+\.\d+)/", $blackfire_version[0], $matches)) {
// and if so, "replace" it, so that we don't install our version - a "provide" would lead the solver to prefer a "real" package instead at least in Composer 1
$replace["heroku-sys/blackfire"] = $matches[1];
file_put_contents("php://stderr", "\033[1;33mNOTICE:\033[0m Blackfire CLI version $matches[1] detected.\n");
} elseif($have_blackfire === 0) {
file_put_contents("php://stderr", "\033[1;33mWARNING:\033[0m Blackfire CLI detected, but could not determine version - falling back to bundled package!\n");
}

$json = [
"config" => ["cache-files-ttl" => 0, "discard-changes" => true],
"minimum-stability" => isset($lock["minimum-stability"]) ? $lock["minimum-stability"] : "stable",
"prefer-stable" => isset($lock["prefer-stable"]) ? $lock["prefer-stable"] : false,
"provide" => $provide,
"replace" => (object) $replace,
"require" => $require,
// only write out require-dev if we're installing in CI, as indicated by the HEROKU_PHP_INSTALL_DEV set (to an empty string)
"require-dev" => getenv("HEROKU_PHP_INSTALL_DEV") === false ? (object)[] : (object)$requireDev,
Expand Down
3 changes: 3 additions & 0 deletions conf/php/apm-nostart-overrides/apm-nostart-overrides.ini
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
agent.auto_launch_proxy = 0
agent.cli_enabled = 0

; Blackfire
blackfire.apm_enabled = 0

; New Relic
newrelic.daemon.dont_launch = 3
newrelic.enabled = 0
Expand Down
62 changes: 62 additions & 0 deletions support/build/blackfire
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
#!/usr/bin/env bash
# Build Path: /app/.heroku/php

OUT_PREFIX=$1

# fail hard
set -o pipefail
# fail harder
set -eu

source $(dirname $BASH_SOURCE)/_util/include/manifest.sh

bin_dir=${OUT_PREFIX}/bin

dep_formula=${0#$WORKSPACE_DIR/}
dep_name=$(basename $BASH_SOURCE)
dep_version=${dep_formula#"${dep_name}-"}
dep_package=${dep_name}-${dep_version}
dep_manifest=${dep_package}.composer.json

echo "-----> Packaging ${dep_package}..."
echo "FYI: Blackfire API reports latest version as $(curl -A "Heroku" -I -L -s https://blackfire.io/api/v1/releases/cli/linux/amd64 | grep -i 'X-Blackfire-Release-Version: ' | sed "s%X-Blackfire-Release-Version: %%i" | sed s%.$%%)"

curl -L "https://packages.blackfire.io/binaries/blackfire/${dep_version}/blackfire-linux_amd64.tar.gz" | tar xz

mkdir -p ${OUT_PREFIX}/var/blackfire/run
mkdir -p ${bin_dir}
chmod +x blackfire
mv blackfire ${bin_dir}/

find ${OUT_PREFIX} -type f \( -executable -o -name '*.a' \) -exec sh -c "file -i '{}' | grep -Eq 'application/x-(archive|executable|sharedlib); charset=binary'" \; -print | xargs strip --strip-unneeded

# these env var defaults we want both during a build (used in the INI futher below) and at boot time
tee ${OUT_PREFIX}/bin/export.blackfire.sh > ${OUT_PREFIX}/bin/profile.blackfire.sh <<'EOF'
# hard-code these two; no need for users to override them
export BLACKFIRE_LOG_FILE=stderr
export BLACKFIRE_AGENT_SOCKET="unix:///app/.heroku/php/var/blackfire/run/agent.sock"
EOF

# PATH should be available both for subsequent buildpacks and on boot
cat >> ${OUT_PREFIX}/bin/export.blackfire.sh <<'EOF'
export PATH="/app/.heroku/php/bin:$PATH"
EOF

# gets sourced on dyno boot
cat >> ${OUT_PREFIX}/bin/profile.blackfire.sh <<'EOF'
export PATH="$HOME/.heroku/php/bin:$PATH"
# we need no config, as everything will be read from the environment
# exception is BLACKFIRE_SOCKET, which we pass in explicitly; the extension uses BLACKFIRE_AGENT_SOCKET instead
/app/.heroku/php/bin/blackfire agent:start --config="/dev/null" --socket="${BLACKFIRE_AGENT_SOCKET}" &
EOF

MANIFEST_REQUIRE="${MANIFEST_REQUIRE:-"{}"}"
MANIFEST_CONFLICT="${MANIFEST_CONFLICT:-"{}"}"
MANIFEST_REPLACE="${MANIFEST_REPLACE:-"{}"}"
MANIFEST_PROVIDE="${MANIFEST_PROVIDE:-"{}"}"
MANIFEST_EXTRA="${MANIFEST_EXTRA:-"{\"export\":\"bin/export.blackfire.sh\",\"profile\":\"bin/profile.blackfire.sh\"}"}"

python $(dirname $BASH_SOURCE)/_util/include/manifest.py "heroku-sys-program" "heroku-sys/${dep_name}" "$dep_version" "${dep_formula}.tar.gz" "$MANIFEST_REQUIRE" "$MANIFEST_CONFLICT" "$MANIFEST_REPLACE" "$MANIFEST_PROVIDE" "$MANIFEST_EXTRA" > $dep_manifest

print_or_export_manifest_cmd "$(generate_manifest_cmd "$dep_manifest")"
File renamed without changes.
81 changes: 6 additions & 75 deletions support/build/extensions/no-debug-non-zts-20160303/blackfire
Original file line number Diff line number Diff line change
Expand Up @@ -40,95 +40,26 @@ bin_dir=${OUT_PREFIX}/bin

dep_formula=${0#$WORKSPACE_DIR/}
dep_name=$(basename $BASH_SOURCE)
if [[ "$dep_formula" != "$dep_name" ]]; then
probe_version=${dep_formula##*"/${dep_name}-"}

echo "Using explicit version ${probe_version}"
else
probe_version=`curl -I -A "Heroku" -L -s https://blackfire.io/api/v1/releases/probe/php/linux/amd64/${series/\./} | grep -i 'X-Blackfire-Release-Version: ' | sed "s%X-Blackfire-Release-Version: %%i" | sed s%.$%%`
cat <<-EOF
!!! WARNING !!!
You're building the generic version of this extension.
API returned version ${probe_version}; it will be built in five seconds.
If you used --overwrite (and no deploy.sh --publish), then this will replace
the existing version of the package under a wrong version number IMMEDIATELY,
even without re-generating the repository, since the archive name is identical:
the repo and new manifest both point to ${dep_name}.tar.gz, but the new manifest
information will not be exposed in the repo until you run mkrepo.sh.
EOF
sleep 5
fi
dep_version=${probe_version}
dep_version=${dep_formula##*"/${dep_name}-"}
dep_package=ext-${dep_name}-${dep_version}
if [[ "$dep_formula" != "$dep_name" ]]; then
dep_manifest=${dep_package}_php-$series.composer.json
else
dep_manifest=ext-${dep_name}_php-$series.composer.json
fi
dep_manifest=${dep_package}_php-$series.composer.json

echo "-----> Packaging ${dep_package}..."
echo "FYI: Blackfire API reports latest version as $(curl -I -A "Heroku" -L -s https://blackfire.io/api/v1/releases/probe/php/linux/amd64/${series/\./} | grep -i 'X-Blackfire-Release-Version: ' | sed "s%X-Blackfire-Release-Version: %%i" | sed s%.$%%)"

curl -L -o probe.tar.gz "https://packages.blackfire.io/binaries/blackfire-php/${probe_version}/blackfire-php-linux_amd64-php-${series/\./}.tar.gz"
curl -L -o probe.tar.gz "https://packages.blackfire.io/binaries/blackfire-php/${dep_version}/blackfire-php-linux_amd64-php-${series/\./}.tar.gz"

mkdir -p ${ext_dir}
tar -zxf probe.tar.gz
cp blackfire-${ZEND_MODULE_API_VERSION}.so ${ext_dir}/blackfire.so
rm probe.tar.gz blackfire-${ZEND_MODULE_API_VERSION}.so blackfire-${ZEND_MODULE_API_VERSION}.sha

agent_version=`curl -A "Heroku" -o agent.tar.gz -D - -L -s https://blackfire.io/api/v1/releases/agent/linux/amd64 | grep -i 'X-Blackfire-Release-Version: ' | sed "s%X-Blackfire-Release-Version: %%i" | sed s%.$%%`
echo "-----> Packaging bin/blackfire-agent ${agent_version}..."

mkdir -p ${OUT_PREFIX}/var/blackfire/run
mkdir -p ${OUT_PREFIX}/etc/blackfire
echo -e "[blackfire]\nserver-id=f1abf3a8-3f85-4743-99b2-97f066c099b9\nserver-token=5ecbc6486e9db6b780a0c0a9ef1e244709e632996fe9105cb9075ab2826944d5" > ${OUT_PREFIX}/etc/blackfire/agent.ini
mkdir -p ${bin_dir}
tar -zxf agent.tar.gz
chmod +x agent
cp agent ${bin_dir}/blackfire-agent
rm agent.tar.gz agent agent.sha1

echo "-----> Packaging bin/blackfire ${agent_version}..."
curl https://packages.blackfire.io/binaries/blackfire-agent/${agent_version}/blackfire-cli-linux_amd64 > ${bin_dir}/blackfire
chmod +x ${bin_dir}/blackfire

find ${OUT_PREFIX} -type f \( -executable -o -name '*.a' \) -exec sh -c "file -i '{}' | grep -Eq 'application/x-(archive|executable|sharedlib); charset=binary'" \; -print | xargs strip --strip-unneeded

# gets sourced on dyno boot
cat > ${OUT_PREFIX}/bin/profile.blackfire.sh <<'EOF'
export BLACKFIRE_LOG_LEVEL=${BLACKFIRE_LOG_LEVEL:-"1"}
touch /app/.heroku/php/var/blackfire/run/agent.sock
/app/.heroku/php/bin/blackfire-agent --config=/app/.heroku/php/etc/blackfire/agent.ini --socket="unix:///app/.heroku/php/var/blackfire/run/agent.sock" --log-level="${BLACKFIRE_LOG_LEVEL}" &
EOF
mkdir -p ${OUT_PREFIX}/etc/php/conf.d
cat > ${OUT_PREFIX}/etc/php/conf.d/blackfire.ini-dist <<'EOF'
extension = blackfire.so
blackfire.log_level = ${BLACKFIRE_LOG_LEVEL}
blackfire.server_token = ${BLACKFIRE_SERVER_TOKEN}
blackfire.server_id = ${BLACKFIRE_SERVER_ID}
blackfire.agent_socket = "unix:///app/.heroku/php/var/blackfire/run/agent.sock"
EOF

MANIFEST_REQUIRE="${MANIFEST_REQUIRE:-"{\"heroku-sys/php\":\"${series}.*\"}"}"
MANIFEST_REQUIRE="${MANIFEST_REQUIRE:-"{\"heroku-sys/php\":\"${series}.*\",\"heroku-sys/blackfire\":\">=2.0.0\"}"}"
MANIFEST_CONFLICT="${MANIFEST_CONFLICT:-"{}"}"
MANIFEST_REPLACE="${MANIFEST_REPLACE:-"{}"}"
MANIFEST_PROVIDE="${MANIFEST_PROVIDE:-"{}"}"
MANIFEST_EXTRA="${MANIFEST_EXTRA:-"{\"config\":\"etc/php/conf.d/blackfire.ini-dist\",\"profile\":\"bin/profile.blackfire.sh\"}"}"
MANIFEST_EXTRA="${MANIFEST_EXTRA:-"{}"}"

python $(dirname $BASH_SOURCE)/../../_util/include/manifest.py "heroku-sys-php-extension" "heroku-sys/ext-${dep_name}" "$dep_version" "${dep_formula}.tar.gz" "$MANIFEST_REQUIRE" "$MANIFEST_CONFLICT" "$MANIFEST_REPLACE" "$MANIFEST_PROVIDE" "$MANIFEST_EXTRA" > $dep_manifest

print_or_export_manifest_cmd "$(generate_manifest_cmd "$dep_manifest")"

if [[ "$dep_formula" == "$dep_name" ]]; then
cat <<-EOF
!!! WARNING !!! If you just deployed using --overwrite and without --publish:
the new manifest points to the updated tarball ${dep_name}.tar.gz;
this tarball will now already be picked up by the existing repository under a
wrong version number. Regenerate repository with 'mkrepo.sh --upload' at once!
EOF
fi

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
# Build Path: /app/.heroku/php

source $(dirname $0)/../no-debug-non-zts-20160303/blackfire
113 changes: 113 additions & 0 deletions test/spec/blackfire_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
require_relative "spec_helper"
require 'ansi/core'

describe "A PHP application using ext-blackfire" do
["blackfireio/integration-heroku", "our blackfire package"].each do |agent|
context "and #{agent}" do
["explicitly", "without BLACKFIRE_SERVER_TOKEN", "with default BLACKFIRE_LOG_LEVEL", "implicitly"].each do |mode|
next if mode == "without BLACKFIRE_SERVER_TOKEN" and agent == "blackfireio/integration-heroku" # blackfire buildpack would error on invalid credentials
context "#{mode}" do
before(:all) do
buildpacks = [:default]
buildpacks.unshift("https://github.com/blackfireio/integration-heroku") if agent == "blackfireio/integration-heroku"
credentials = {
"BLACKFIRE_CLIENT_ID" => ENV["BLACKFIRE_CLIENT_ID"],
"BLACKFIRE_CLIENT_TOKEN" => ENV["BLACKFIRE_CLIENT_TOKEN"],
"BLACKFIRE_SERVER_ID" => ENV["BLACKFIRE_SERVER_ID"],
"BLACKFIRE_SERVER_TOKEN" => ENV["BLACKFIRE_SERVER_TOKEN"],
}
if mode == "explicitly"
# ext-blackfire is listed as a dependency in composer.json, and a BLACKFIRE_SERVER_TOKEN/ID is provided
@app = new_app_with_stack_and_platrepo('test/fixtures/bootopts',
buildpacks: buildpacks,
config: credentials.merge({ "BLACKFIRE_LOG_LEVEL" => "4"}),
before_deploy: -> { system("composer require --quiet --ignore-platform-reqs 'php:*' 'ext-blackfire:*'") or raise "Failed to require PHP/ext-blackfire" },
run_multi: true
)
elsif mode == "without BLACKFIRE_SERVER_TOKEN"
# ext-blackfire is listed as a dependency in composer.json, but a BLACKFIRE_SERVER_TOKEN/ID is missing
@app = new_app_with_stack_and_platrepo('test/fixtures/bootopts',
buildpacks: buildpacks,
config: { "BLACKFIRE_LOG_LEVEL" => "4" },
before_deploy: -> { system("composer require --quiet --ignore-platform-reqs 'php:*' 'ext-blackfire:*'") or raise "Failed to require PHP/ext-blackfire" },
run_multi: true
)
elsif mode == "with default BLACKFIRE_LOG_LEVEL"
# ext-blackfire is listed as a dependency in composer.json, and BLACKFIRE_LOG_LEVEL is the default (1=error)
@app = new_app_with_stack_and_platrepo('test/fixtures/bootopts',
buildpacks: buildpacks,
config: credentials,
before_deploy: -> { system("composer require --quiet --ignore-platform-reqs 'php:*' 'ext-blackfire:*'") or raise "Failed to require PHP/ext-blackfire" },
run_multi: true
)
else
# a BLACKFIRE_SERVER_TOKEN/ID triggers the automatic installation of ext-blackfire at the end of the build
@app = new_app_with_stack_and_platrepo('test/fixtures/bootopts',
buildpacks: buildpacks,
config: credentials.merge({ "BLACKFIRE_LOG_LEVEL" => "4"}),
before_deploy: -> { system("composer require --quiet --ignore-platform-reqs 'php:*'") or raise "Failed to require PHP version" },
run_multi: true
)
end
@app.deploy
end
after(:all) do
@app.teardown!
end

it "installs Blackfire" do
if agent == "our blackfire package"
expect(@app.output).not_to match(/Blackfire CLI version \d+\.\d+\.\d+ detected/)
else
expect(@app.output).to match(/Blackfire CLI version \d+\.\d+\.\d+ detected/)
end

if mode == "implicitly"
expect(@app.output).to match(/Blackfire detected, installed ext-blackfire/) # auto-install at the end
else
if agent == "our blackfire package"
expect(@app.output).to match(/- blackfire/)
else
expect(@app.output).not_to match(/- blackfire/)
end

expect(@app.output).to match(/- ext-blackfire/)

if mode == "with default BLACKFIRE_LOG_LEVEL"
expect(@app.output).not_to match(/\[Debug\] APM: disabled/) # this message should not occur if defaults are applied correctly at build time
else
expect(@app.output).to match(/\[Debug\] APM: disabled/) # extension disabled during builds
end
end
end

['heroku-php-apache2', 'heroku-php-nginx'].each do |script|
# without log level info, we will not see the messages we're using to test any behavior
# but we need to assert that no info is printed at all in this case
it "does not output info messages during startup with #{script}", if: mode == "with default BLACKFIRE_LOG_LEVEL" do
out = @app.run("#{script} -F conf/fpm.include.broken") # prevent FPM from starting up using an invalid config, that way we don't have to wrap the server start in a `timeout` call
expect(out).not_to match(/\[Info\]/) # this message should not occur if defaults are applied correctly
end
it "launches blackfire CLI, but not the extension, during boot preparations, with #{script}", if: mode != "with default BLACKFIRE_LOG_LEVEL" do
out = @app.run("#{script} -F conf/fpm.include.broken") # prevent FPM from starting up using an invalid config, that way we don't have to wrap the server start in a `timeout` call

out_before_fpm, out_after_fpm = out.unansi.split("Starting php-fpm", 2)

expect(out_before_fpm).to match(/blackfire Reading agent configuration file/) # that is the very first thing the agent prints
if mode == "without BLACKFIRE_SERVER_TOKEN"
expect(out_before_fpm).to match(/The server ID parameter is not set/)
else
expect(out.unansi).to match(/blackfire Waiting for new connection/) # match on whole output in case it takes a bit longer to start <up></up>
end
expect(out_before_fpm).not_to match(/\[Warning\] APM: Cannot start/) # extension does not attempt to start on `php-fpm -i` during boot
expect(out_before_fpm).to match(/\[Debug\] APM: disabled/) # blackfire reports itself disabled (by us) during the various boot prep PHP invocations

expect(out_after_fpm).not_to match(/\[Debug\] APM: disabled/)
expect(out_after_fpm).to match(/\[Info\] APCu extension is not loaded/)
end
end
end
end
end
end
end
1 change: 1 addition & 0 deletions test/var/log/parallel_runtime_rspec.heroku-18.log
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
test/spec/blackfire_spec.rb:239.20765019800092
test/spec/bugs_spec.rb:45.368704143
test/spec/ci_spec.rb:274.321140925
test/spec/composer_spec.rb:30.616864848999995
Expand Down
Loading

0 comments on commit 004442d

Please sign in to comment.