OAuth Client Credential Token APIs

So I’ll probably get hugely humiliated by writing this post – but then again, how do we learn without failing…

I today had the chance to watch DJ Adams run through one of his #HandsOnSAPDev live streams: (it start’s 3mins 30secs in if you want to watch)

It was an interesting watch and makes me long for those days when I could just play with APIs marked beta and hope to get away with not having to do a massive code re-write several months later.

Anyway – during the episode, DJ was using the “standard” OAuth Client Credentials flow and I had to ask myself – “Why is this any more secure than just using basic authentication?”

I tried to put the point to DJ – but there’s only so much one can do in a chat window, hence this post.

I have since tried to do a little research on the topic – and found this rather good Stack Exchange discussion on the topic:

https://softwareengineering.stackexchange.com/a/297997/369892

Both the question and answer made a lot of sense – read if you like – but I’ll work through the points – in my own style – which is to illustrate with bad drawings.

OAuth Client Credential Flow explained with bad drawings

In the OAuth Client Credentials flow – one system (Bob, our client) gives another system (Dave, our authorisation server) his special secret key.

Bob uses his secret key to authenticate himself to the authorisation server. In the example DJ worked through authentication was in form of an authorisation header with <clientid>:<clientsecret>. (And a body that contained the user’s username and password – this is useful for API’s that need to pickup a given user’s credential.) Note that nothing here is encrypted beyond the usual transport encryption. I’ve seen many implementation of this process where the username isn’t actually needed because the particular client id and secret are associated with a particular system user. (I’ve never seen any other one where the user’s password is needed – noting these API’s are beta!).

Once Dave (our authorisation server) gets our secret, he checks it is still valid (kinda like checking a password… no wait, exactly like checking a password) and then gives us a limited lifetime token to use the API. In the example DJ worked through it also checked the username and password – strange, but hey why not? (Other than it being a TERRIBLE idea that any server should need to store a user’s password as well as client id and secret!)

Now, according to the OAuth standards, Bob could have asked for the token he picked up to be scoped to only allow certain access. But because Bob is a little bit lazy and Dave doesn’t insist that he asks for scope, Bob never does. If you got to the oauth.com website and check out the client credential flow (https://www.oauth.com/oauth2-servers/access-tokens/client-credentials/_ you’ll even see they mention that hardly anyone ever uses scope in the flow.

Your service can support different scopes for the client credentials grant. In practice, not many services actually support this.

OAuth.com

Plus Bob likes reducing his interactions with Dave to make things faster, so one token to rule them all is far easier and more generic. Bob might be a programmer (if he weren’t a system… stick with the metaphors people!)

Bob now has his limited lifetime access token he can use to authorise the API interactions. So he goes to make call to the API server.

Imagine Bob’s surprise when talking to the API server it looks very very much like Dave. But it’s not Dave, it’s Fred the API server. In DJ’s example the authentication server was accountblah.authentication.region.hana.ondemand.com and the API server was accounts-service.cfapps.region.hana.ondemand.com. Slightly different names – and actually resolve to different IP addresses too! But if I look at the SuccessFactors implementation of this similar token logic – both sit on same server (from an external view – who knows or cares what happens internally). Anyway – Bob uses his token to request some data from Fred.

Fred then goes away and checks that the token is valid. When the token is sent over to Fred, it’s not encrypted in any way or signed with a special key or anything. In DJ’s example it was just in the Authorisation header as “Bearer <access token>”. The security of this exchange was relying on the transport encryption – just like the original request to get the access token.

Fred may well be wondering if Bob is ever going to send him a request he doesn’t have scope for, might need to have a chat to Dave about that… But he validates that Bob has a token that is still valid and that is valid for the requested action (get list of sub-accounts for example.)

So what makes this more secure?

In the exchange I just documented, I cannot see how taking the extra step to pick up an access token to call Fred has made the exchange between Bob and Fred more secure… The only things I can think of are:

  • Conversations to Dave (the authentication server) are treated more seriously, we take extra special care to not record them or allow anyone to snoop on them because the client secret is long lived (like a password).
  • Possibly means that if we take less care that conversations with Fred leak then the impact will be short lived due to the token expiring sooner.
  • yep – that’s about it.
  • can’t think of anything else.

Frankly – for the increase in hassle I’m not seeing an ROI on securing my API calls. Especially as for many implementations of this sort of logic the token API runs on same server as the main application API. (Dave == Fred)

What makes this useful then?

This is different thing – and goes to identity rather than authorisation. With the client credential approach I can config my calls to the API server to be treated as if one of the system users is making the call and not a generic API user. I have one password that I use to get access tokens that allow me to “pretend” to be any user I want to be for the purpose of fetching/updating data via the API. This is something that I would use all the time in SuccessFactors* – it lets me query data using the user’s permissions. Very useful! SAPCP is set up to do this. I believe this is how, for example applications running on SAPCP can use OAuth Bearer destinations to access API calls as per the logged in user – even though the user is not logged in to that remote application. We can’t do lots of client side SSO, because browsers have gotten wise to applications doing SSO to remote system inside frames (SSO to a remote server requires JS running on different domains generally and falls foul of Single Origin restrictions). So solutions like SAP Cloud Portal and now SAP WorkZone use the SAPCP destination service to call OAuth Client Credential flows to get access tokens as per the person that is logged into their solution. Obviously this requires trust set up – which is the the client id and secret.

So Claire (an actual person, not a system for once) comes along and asks Bob (the system) for some data that’s on the system that is Fred (are you lost yet?). Bob asks Dave for an access token that will allow him to ask Fred for stuff on behalf of Claire. Dave, being ever so obliging and having verified the client id and secret, then gives Bob a temp pass to pretend to be Claire when speaking to Fred.

Okay if you’re following that, you’re doing well. But really – that’s why we use these credentials. It has nothing to do with security and everything to do with making life easier for system to system communication when pretending to be other people.

* Now to explain that aster a bit up the page. I would possibly use this logic in SuccessFactors, but I don’t because it requires that the user that “Bob” is pretending to be needs to have API access or Fred refuses the call. Giving all users API access is not a good idea at the moment in SuccessFactors because of the way that certain fields tend not to be hidden or controlled in API access compared to front end access.

Summary

So, to summarise, I cannot see any real security benefit to using OAuth Client Credential flows over Basic Auth unless you are looking to distribute your development spend on making one area of your codebase more secure than others. Even then it’s not that much better. If you’re able to intercept and abuse a basic authentication flow, you’d be just as likely to be able to intercept and abuse an OAuth Client Credential flow. Indeed because organisations tend to use the Client Credential flow as per the example DJ had (with the credential applying to a given user) or like SuccessFactors does, it actually open up a whole new security issue… It’s not just one “user” that might have their credentials breached, it’s anyone that the system is allowed to impersonate.

Okay – go at me – I’ve missed something – else we wouldn’t be using OAuth Client Credentials for the sorts of API calls that DJ was making in the video.

I note there are many other OAuth flows and some of them are much more secure – they use PPK encryption to ensure that messages are signed and headers never could leak credentials like Basic Authentication can. But the client credential flow – hmmm this one, in the use case where it’s a single user, not “impersonating” anyone – there’s no benefit over Basic Auth and another communication round trip to have to deal with.

It’s a kinda Magic!

Okay – so I’m supposed to be doing something else right now, but a) I’m procrastinating, b) I just got off a call with a whole bunch of SuccessFactors integration “experts” and I want to show that I haven’t completely lost touch! And c) I’ve finished Witcher 3 Wild Hunt and all the add ons and played far too much Civ6 so I really needed to do something else (which wasn’t the thing I was avoiding doing in part a).

So, I thought I’d share a teeny tiny simple integration that I helped a customer build the other day. Yep – I didn’t build it – I helped the customer do it (a functional SuccessFactors admin) via a web meeting.

Updating data in SF WITHOUT using a workflow just simple integration

It came about because we had just updated from old and honestly less than perfect Boomi integrations to newer (and still not perfect, but better) SAP Cloud Platform Integration based integrations. One of the things that was “fixed” was that leave records in SF and SAP were now being kept in sync. So an old process that they had running in SAP that was updating SAP leave records without updating the SF record was no longer working. Well, truthfully, it did work, right up until the next sync run happened and then the record got updated back to the SF state.

Requirement(ish)

The functional requirement was based on a process this customer has. They allow managers to enter a “unexplained” absence type for employees into SuccessFactors in the case that an employee is a no-show for a given day. When the employee returns to work, they have the option to change that absence type to a personal leave (sick leave or carers leave) or mark it as annual leave, or however it needs to be handled. But if they don’t update the record within 14 days the leave type is automatically changed to personal leave without medical certificate.

Design

On the basis that “Any sufficiently advanced technology is indistinguishable from magic”

Badly%20drawn%20whiteboard%20design%20for%20solution%20with%20MAGIC%20in%20sparkly%20colours%20whenever%20integration%20mentioned

I present how we got it to work again in my bad whiteboard drawing style. NB payroll is shown as a little black box because you really don’t want to look in there, and I clearly do not have enough skills with drawing to represent Pandora’s box.

So I’d like to take the time to explain how we got the “Magic aka Integration Center” bit to work. (Yes I know that there are two spellings for Centre/Center and really I don’t mind. If I did I would blame the US, but they’re in a world of hurt right now, so let’s be nice).

Making the Magic happen

In Integration Center we created a new integration:

Chose the “more integration types”

And then chose Scheduled SuccessFactors to SuccessFactors OData v2

Chose the Employee Time entity

Then simply dragged and dropped the external code from source to destination, fixed the time type to 0200 (Personal Leave)

And set the operation to “Update/Merge” (which is a personal bugbear of mine since the HTTP operation should really by “Patch” not “Merge” but that’s OData v2… <sigh>)

Now if we were to run this now it would transform all leave record in the system to time type 0200 – which would be bad… (with great power comes great responsibility Peter!)

So next up let’s restrict which records should be selected…

I’ve only set up one “Unexplained” absence in the system – has time type of 6666

Now we only want to update entries that are time type 6666 and creation date is 14 days in past (I’ve used 14 minutes for this example because waiting 14 days seems a bit excessive for such an unpolished blog post)

Note that date filters have a very useful “relative date” option.

Now jumping back to the configure fields screen and you should see only one entry… (NB if you choose created DATE rather than created DATETIME, just taking 1 min off actually takes you to previous day so be careful what you choose!)

Now if I wanted, I could schedule the job to run daily:

Then just set the schedule and go for it!

But to be sure it’s working, I can actually just run it in preview:

And it works!

Entry updated to personal leave

So simple, yet so powerful!

Usual disclaimers apply – and especially so given the whole Spidey superpower unleashed here. You can really muck up your data doing stuff like this… DO NOT TEST IN PRODUCTION!

Enjoy. Don’t be afraid to give it a go! I’d advise, just go out try it. You learn so much more that way rather than asking other people to figure things out for you!

“How soon is now?” or Thanks for listening

With many thanks to Meg Bear for listening and engaging and tweeting this video to question about how soon will the unofficial-official announcement that SuccessFactors customers will get more than one non-prod IAS instance.

So I was staying up late (12:30am online meeting) to join a partner IAS/IPS Q&A/briefing session, when I looked through the latest doco that SAP SuccessFactors had released about the connection of SuccessFactors and IAS tenants and I found something that I swear wasn’t there before…

The above can be found at: – SuccessFactors to IAS Tenant Mapping Guidelines

I did quickly check with Meg Bear that I wasn’t dreaming and this was real and she confirmed – yes this is now “public” and was real.

So yes Chris you can have “one more”

It’s really heartening when you go to the effort to put a case for a change of position to an organisation like SAP SuccessFactors and they listen, engage and then implement change. I’m really happy that customers are going to get another IAS for their non-prod environment, it makes so much sense. However, I’m even happier that this clearly shows how the team at SuccessFactors are willing to listen and work with the community to make the product better.

Win fricking win!

Individually, we are one drop. Together, we are an ocean. Ryunosuke Satoro
Community FTW

How to break a shared authentication solution

Okay, even with the number of diagrams in my last post, there was still some confusion. So, I’m going to try to make it totally clear. Emphasis on “try”.

Regression Testing isn’t just about regression testing one part of a solution

Firstly, I got some feedback from SAP Support when I tried to raise this issue with them. They didn’t seem to understand my concern that I tested the whole recruit to hire to fire employee life cycle every time that SAP released new functionality for SuccessFactors in the preview environment before it got released to the productive environment. This was because:

“Since IAS productive and test instances are of the same version, there is from IdP perspective no difference at all.“

…

“As already mentioned above: IAS test- and productive instances are of the same release. There is no such thing as a preview environment for IAS.
Thus again one single IAS tenant will be fully sufficient to handle the described scenarios from IdP perspective“

I’m not sure how I can make this clearer. But, yes, I do want to test that during the preview release update period the changes that SuccessFactors make do not impact the provisioning and login processes that I have configured in IPS/IAS. I also want to ensure as well as ensure that these are working in the non-prod environment so that if I push up a change to fix a release impact, it doesn’t break stuff. I’m confused how the Identity services teams seem to think their solutions works in isolation! If it isn’t for the systems that use their services and provide the user details the solutions aren’t worth anything!

How does this all look if we try to connect it up?

Okay – here’s a relatively complex diagram. It shows how you’d have to wire up the provisioning and authentication trust when using more that one SuccessFactors system with a single IAS.

But there is the slim possibility it could work!

And it could work provided that you:

  • Did some configuration in the IPS and IAS that SuccessFactors has not at all been documented for customers
    • that populated userid into different custom attributes of the IAS user record per system
    • then used that system specific field in the assertion sent to the different systems.
  • AND – never maintained different employee attributes for the same email address in the two systems (if you don’t want the system to get hella confused.)

Unfortunately that’s about as likely as me winning the Powerball lottery. Whilst the first point is terribly technical and pedantic (therefore will be loved by half the people I know and hated by the other half) the second point pretty much means this is never going to happen. The reason for the second restriction is that you cannot have the same email address against multiple “user” records in IAS. Whilst through some technical wizardry I might get the same record to point to two different system ids, whether it is active or inactive is a non-system specific piece of user data. What name that user has is a non-system specific piece of user data.

So assume I do have the same email address assigned to an employee in both systems A and B. Terminating that employee in system A will cause a delta in the employee record that will get picked up by the IPS and deactivate the user in the IAS. Even though they should be active to be allowed to log in to system B, they won’t be able to as they are now “inactive”.

It is possible that customers will just decide that they will ensure consistency between user records in different systems, use same name, have both active, have both inactive, but I very much doubt it.

Likewise not using the same email address for employees in both systems is going to be hard (not to mention hard to track if it mistakenly happens).

Would be nice, but unlikely

In my previous post, I assumed that the previous scenario (especially as it involves undocumented configuration) would not be customers’ default. There I assumed that customers would use the default configuration that is deployed when a customer implements the SuccessFactors IAS “upgrade”. That then allows for all sorts of mischief!

How to abuse authentication when you control the data it is based upon.

Let’s have some fun. Assume that default IPS configuration is being used and the employee records from systems A and B are both trying to update the IAS user master record. Assume system B contains sensitive payroll data (for example it is a copy of productive system). Only Annie Admin has the roles in system B to see this data. I, Chris Creative, have access to system A where I’m doing some project work. I have a role where I can hire and fire employees (tends to happen in HR systems!)

  • I firstly terminate an employee with Annie Admin’s email address in system A. If there isn’t one, I’ll just hire her then term her.
  • This will trigger the IPS to update the IAS with the user with Annie’s email address as inactive so Annie can’t log into either system and stop my nefarious fun.
  • Then I hire a new employee that has the same personnel number as Annie had in system B and put a newly generated email address (that I have access to) against the employee’s record
  • I get emailed by the IAS that I have a new user record set up 🙂 what a cool system! I set/reset the password.
  • Now I can use this email address to access system B and it thinks I’m logging in as Annie! Which is awesome as when they try to trace who downloaded all the payroll details from the system the audit logs are going to clearly point to Annie’s user! (Okay yes a bit of further checking and someone might see that it was my IP address, but I can hide that using a VPN, the audit logging in IAS is hard to get to (API access only) so going to be hard for them to find the random gmail address I made up and trace it.
  • I access a bunch of data I’m not supposed to
  • I go back to system A, change the email address of the fake Annie back to her email.
  • She gets reactivated
  • I just got away with accessing a system that I had no rights to access because I had rights to another system that was provisioning the same authentication system.

Are you worried yet?

Can you see now why I’m a little worried about this setup? This is why I also assumed that to prevent this sort of thing happening, most organisations if forced down the path of using just one IAS will choose the most secure SuccessFactors system to provision user data to the IAS. But that, as I wrote about in the previous post will cause a bunch of other problems.

Simple solution

In case it’s not obvious enough, there is a simple solution. Provide another IAS instance per SuccessFactors instance that a customer has.

Please sir, can I have one more?

So, firstly, at this moment in time it feels a little petty writing about this. The world is just starting to realise how huge a problematic space we are in with COVID-19. So I wish all of you out there reading this, health and the wisdom to look after yourselves and others. So, now back to me being needy.

tldr;

SAP SuccessFactors are only offering customers one non-prod IAS system. One non-prod IAS is not enough. Read more to understand what that is and why I think this is a mistake.

A new service to support new functionality

Recently SAP SuccessFactors announced that they are going to move all customers to a new platform authentication solution which uses SAP Cloud Identity Authentication*.  This new background service is a requirement to use the new People Analytics that use SAP Analytics Cloud and is also required to use new internal facing Career Site builder functionality. In short, to take advantage of some of the newest and coolest looking features of SuccessFactors, you’re going to need to implement this!

*There’s a handy overview video (strangely enough not behind SuccessFactors community wall) that explains in detail what SAP is doing, it’s a little more in depth than the detail I provide in this post, it’s not bad although very detailed.

Why are SAP SuccessFactors doing this?

Honestly, this is a good move. Authentication is a common problem in all cloud based solutions, so why not have a single service that can be leveraged by all SAP solutions to solve this? Customers get the benefit of multiple development teams within SAP all pooling to produce a better product, SAP gets the benefit of reducing cost of having to maintain and upgrade their authentication solutions for each cloud product. Win – Win!

balance of cost to SAP vs benefit to customer, for once, both sides of the scale seem to be wanting to go up!
Not often the balance seems to favour both sides!

It’s not yet a requirement for customers to migrate to this solution (and yes for many customers this will require some work to implement), but in order to take advantage of several new SAP SuccessFactors functionalities, customers are going to have to move. So I’m pretty sure that SAP will get close to its goal of migrating by end of year (although possibly with a covid related bump in that progress).

“Our goal is to have customers migrated to SAP Cloud Identity Authentication (IAS and IPS) by the end of 2020. This is not a “forced” migration, we are just encouraging customers to migrate.  At some point all customers will need to be using this authentication method, that date is not yet determined. “

from SAP SuccessFactors Community, Platform Resources Blog

However, in going through the process to migrate some of my own systems, I’ve discovered a bit of a problem. I’ve tried to raise this multiple times, but so far, I just don’t seem to be able to clearly articulate why this problem is so important. So, I thought I’d try putting together a simple blog post with LOTS of pictures. Hopefully more pictures means easier to understand and then we can get somewhere!

Background – How does it work?

Okay, before diving into the deep end, it’s probably worth trying to articulate what on earth this SAP Cloud Identity Authentication is, and why I’m so happy that it’s coming.

Simple how to log on process

So, here’s how it works in general.

  1. Happy SuccessFactors user uses their computer to
  2. Access SuccessFactors website, which
  3. Redirects them to their own company’s identity provider (IDP) which asks them to log on (or possibly realises they are already signed in and does single sign on. Which then
  4. Sends them back to SuccessFactors having verified that they are a valid user and they can access their SuccessFactors system, so
  5. The Ecstatically Happy SuccessFactors user is even more happy!

Pretty simple really! There’s some technical stuff like SAML2 assertions and stuff that happens – but generally the experience and security are what matters.

Importantly, SAP SuccessFactors aren’t really aiming to change any of that experience, there just a small technical difference. In between steps 3 and 4 there’s going to be a little more going on under the hood.

Steps 3 and 4 of our previous diagram are replaced with 4 other steps. Quite simply SAP Cloud Identity Authentication Service (IAS) becomes a middleman to/from SuccessFactors and the Corporate IDP.

  1. SuccessFactors sends unauthenticated requests to IAS.
  2. IAS redirects request to IDP for authentication.
  3. IDP tells IAS that user is identified and correctly authenticated
  4. IAS tells SuccessFactors that user is identified and correctly authenticated.

There are a few benefits here. The SAP Cloud IAS is set up to be maintainable by customers. Whereas setting the details of a corporate IDP into SuccessFactors was/is restricted to SAP and certified partners, using the IAS is something that most customer’s technical teams should be able to manage. (I’ll caveat that with note that I expect that most customers will probably get their implementation partner to guide them through the initial setup. But at least it is something that customers can do, if they want to.) SAP SuccessFactors automagically set up the link between SAP SuccessFactors and the IAS.

There are many options to be able to configure the IAS to do some nice things, setting up rules for access, making login screens pretty, etc. It’s a good tool.

It is important, however, to understand that the IAS isn’t just a simple pass-through proxy service for authentication, it stores a list of all active users. A user MUST exist in the IAS in order to be able to log on to SuccessFactors.

So, what’s the problem?

Well so far everything seems hunky dory, no? Well, that’s probably because I’ve just talked about the productive use of the solution (which to be fair is what most people care about.) However, for many customer non-productive environments (and even some productive ones!) a corporate IDP and single sign on is not used.

Simple password-based access not using a corporate IDP
  1. Uses browser,
  2. Logs on with username/password (stored in SuccessFactors)
  3. Is happy

This scenario is updated also with IAS in the picture.

Can we just assume that you’re as tired of reading numbered lists as I am of typing them? No? Okay – just to be complete:

  1. User bounces to their laptop…
  2. Uses browser to access SuccessFactors
  3. Which redirects them to IAS which asks them for username and password (as no corporate IDP configured)
  4. IAS tells SuccessFactors that user successfully identified and authenticated.
  5. User has great time using SuccessFactors without SSO or a corporate IDP

There is some great functionality that can be leveraged here too! SAP Cloud IAS can implement multi-factor authentication – pretty damn good to have that available without having to use corporate IDP!

Sidebar – there is an IPS too

Yes, sorry more terminology! The way that the IAS works it needs to have a list of users provisioned into it. It doesn’t just use the list of employees that are in SuccessFactors, these need to be populated into the IAS.

  1. The IPS reads SuccessFactors regularly to find any changes to employee/user records
  2. It passes any details if finds to the IAS which updates its list of active users for SuccessFactors (and potentially any other system that authenticates against it).

Now that might sound like a retrograde step – it means that we need to provision (yep IPS stands for Identity Provisioning Service) users from SuccessFactors into IAS on a very regular basis. But, realistically, there are very few productive use scenarios that require a new employee to have access to the SuccessFactors system instantaneously, a delay of an hour is normally manageable. And there are ways to make that sync happen faster if needed.

Again, this seems all to be good news! There are a few minor niggles. For instance, the IAS must keep a list of usernames and email addresses in order to allow it to function. One additional restriction is that all users must have unique email addresses (you can’t set all the non-used users’ emails to dummy@dummy.com!) Again this doesn’t seem to be much of a problem. Until one considers that many customers have more than one non-productive system.

There can be only one!

I was so tempted to put a Highlander meme in there… But I didn’t because it might distract from the seriousness of this next bit – which seems to be the official SAP position at the moment.

” SAP is offering free [IAS] licenses to SuccessFactors customers for the purpose of logging-in to SF; they will receive one production and one test instance (by region).  “

SAP SuccessFactors IAS FAQ, highlighting and info in [brackets] added by me

SAP will only provide customers with one IAS tenant for their productive environment and one for their non-productive environments.

Why is this a concern?

This seems to be the bit that I’m having trouble explaining to people, so I hope the following diagrams and explanations help.

For many SuccessFactors Employee Central (Core HR) customers they have been provisioned by default with three instances. Customers can use them however they like, but I tend to advise people to do the below:

landscape of preview, non-prod and productive SF instances

Note that these systems are very often linked to payroll environments that reflect similar levels of detail to the productive environments. As such, the levels of control of access are very important. This is personal and private information and needs care and attention!

However, very often one of the systems will have anonymised data, or even just made up data, and be used for prototyping and building solutions. Generally in this system users may be given access to areas that they don’t normally have access to. Training and/or testing may take place. The key thing is that users may be running scenarios with different access levels than normal.

It’s important to note that whilst there is clearly an overlap in user records between all three systems, there are plenty of users in both non-productive environments that don’t overlap in normal usage of these systems.

Again, so why is this concerning?

Well, to reiterate, SAP have said they will only provide as part of customers’ existing subscriptions, two IAS instances. One for productive use and one for non-productive/test environments.

The non-productive IAS must provide authentication services for both non-prod environments. However, the feed that populates the list of users into the IAS can realistically only come from one of the two environments.

I’d better make QAS my source of user data then

Due to security concerns for data access, I must then make the source of user data for the IAS the QAS system, where I have locked down user access. Otherwise, if the source of the user data was the test system, then a test user could update a user in the test system records to have details that could allow themselves to log on to the QAS environment and see data that they were not supposed to see. That can’t be allowed!

Real email addresses at a premium for testing, double handling.

It will be quite some hassle to maintain all the Test system users in the QAS system, but it could be done. However, it may be very difficult to test scenarios that rely on real email addresses. Because the email address must be unique in the IAS, it won’t be possible to swap emails around easily. This becomes especially problematic when email address is used to authenticate users to downstream systems (particular cases where I have seen this occur are SSO into SAP ERP systems, where email address is used as the unique identifier.) Changing an email address to allow testing of a process may mean creating a new employee in the QAS system and reassigning email addresses there in addition to updating the test system.

But hold on, what about regression testing?

However, during the 6 monthly release cycle, there is a need to test all the new SAP SuccessFactors functionality. This needs to be done in the preview environment. So perhaps during this period I lock down all access to the preview environment, make user access as per the QAS system? Then I switch my IAS source to be the preview environment? This has some problems associated with it too. It really restricts how I can carry out my regression testing and who I can use to do it. But sure as a very sure thing, I will want to test out user provisioning and definitely want to check that the configuration I’m using to populate the user list into my IAS from SuccessFactors hasn’t been impacted by the half yearly release.

So where does this leave us?

Very soon, with the release of SuccessFactors embedded analytics (People Analytics), more and more customers are going to want/need to implement SAP Cloud IAS within their SuccessFactors environments. I would imagine that due to the overheads that I described that many customers would opt to purchase an additional subscription for another non-productive IAS instance. If I think of the cost/overhead of:

  • maintaining all non-productive users in pre-prod (imagine having to hire someone in the pre-prod environment just so you could test a hire in the test environment!)
  • Separating out test system users/data from pre-prod environment to ensure regression testing payroll changes.
  • Migrating IAS and IPS (tool used to synchronise user record to IAS) configuration from one system to another to enable release testing, then flipping back if urgent production support testing needs to occur.
  • Restricting release regression testing to only those users that have pre-production access and only to their regular roles.

Then I think I can see the cost benefit of purchasing an additional IAS system to handle that. But I really don’t want to be forced into paying additional subscription to handle a scenario that works just fine right now.

It seems that the balance has shifted and not it a good way for customers.

Given that when I raised this point in SuccessFactors community site I was told that:

…By design, you don’t need a 1 to 1 relationship between SF instances/tenants and IAS.  

IAS is designed like any other Identity Management product to handle logins for many different systems. Customers don’t buy a new copy of Ping Identity or Microsoft Azure for every application they use it for.  Your IAS configuration will control which SF instance users log into. 

Additionally, having the 1:many approach makes it a lot easier to manager [sic]. If you had multiple copies of IAS/IPS you would need to figure what is connect to what every time you want to manage anything. When you re-use the same IAS for many applications, you only have to configure one time and then re-use them.  ie…Password policy settings, Corporate Identity Provider connections etc. The corporate IDP is a huge one since you have to work with your internal SSO folks any time you change anything there. 

I think there is a disconnect between how some SuccessFactors product managers think customers are using their product and how it is being used. Many customers do not use SSO into their non-productive environment by design!

Next steps

I really hope that by spending the time to put this post together I can raise some awareness of this problem before it becomes a bigger issue for customers. Ideally, I’d love SAP SuccessFactors to re-evaluate their stance on providing only one IAS instance for all non-productive SuccessFactors instances. It should clearly be (in my not so humble opinion) one IAS per SuccessFactors instance. (on a technical note, happy for just one non-prod IPS). Let’s see. If you’re reading this and you have some influence with SAP SuccessFactors it would be great if you let me know what you think and perhaps let others know too.

Last thought on solution parity

Finally let me leave you with one last diagram/thought…

I love working with the SAP SuccessFactors software and I think that the IAS is some great functionality. However, no new functionality that replaces existing functionality that a customer has should ever require them to purchase additional subscription to retain parity with their existing solution.

P.S. there’s a follow up post to this one to clarify a few things – How to break a shared authentication solution where I put my evil hacker hat on and give and example of why this could be very ugly for an organisation.

SuccessConnect 2019

Just some very brief highlights and musings from the lounge in LAX whilst I wait for my flight home to Melbourne.

This was the eXperience SuccessConnect, there were X’s everywhere! The amount of airtime that Qualtrics and experience management got may well have equalled the amount of time that the SuccessFactors product was discussed.

My SuccessConnect really kicked off with  the Monday partner briefing and meeting up with Sylvie Otten and introduced to Micheal Grimm from Ingentis. Was good to start off the event with a good moan about the state of CF vs Neo when it came to SuccessFactors extensions. Sorry Sylvie – I earned my namesake.

At least she was smiling in the photo!

 

Next up was the partner briefing – after having made the mistake of sitting near the front last year at it being the most boring multiple hours of the conference, I sat at the back so I could sneak out rather than pass out with boredom.

I was wrong.

David Ludlow completely creamed it with his update on roadmap and recent changes. That was probably the best 20mins of the entire conference. Wish I had sat nearer the front! Although have discovered that taking photos of slides in SAP’s newly preferred dark theme really doesn’t come out well, the black comes out browny grey – doesn’t look so cool… 🙁 But I got the slide deck – gold!

There were many cool things in there, but the new UI being proposed for the user landing page was probably the most exciting bit – which is why David didn’t say much about it, but left that for Amy the next day at the keynote.

Lots of debate about this new conversational AI based UI. With the biggest one being, there better still be menus and some way i can add a link to view my payslips, and apply for leave  with one click without having to type in “view my payslip”.

We’ll see how it develops as looks like planned delivery 2020 – but I’ll bet on it not being GA before next SuccessConnect. Perhaps beta with a couple of customers… lets see next year.

After David’s excellent presentation I got to catch up with the team of analysts and media folk who’d had a briefing that day. Josh Greenbaum was holding court and had some interesting comments around SAP’s strategy with payroll, but the wine and food was excellent and I left with some of the folks there to go to the exhibition show floor and do what I was supposed to be at the conference for (not chin wagging with analysts!)

I was here at SuccessConnect to sell a SAP CP extension for SuccessFactors that me and my team have built – it provides EHS functionality embedded into SuccessFactors. It’s pretty cool! But I would say that!

 

I have discovered that I sell much better with a glass of wine in my hand…

After that it was to Jewel nightclub to the Kronos and Rizing party (not sure I really should have continued drinking, next day was hard work!) But did get to meet up with Sherryanne Meyer which was nice.

 

The next day was lots of water and the keynote.

Image

Have to say, I love the moments that matter idea. as in consider where you are putting your effort in making certain processes work better. It’s probably much more important that the initial day one experience for an employee is better than the process they have to use to apply for leave. They might apply for leave hundreds of times, but they only have one day one and it will be a much bigger impact. Think about the return to work process for return from parental leave, make that a great experience don’t worry so much about timesheet entry….

However I heard the term FAR too over used during the conference and used outside of the context i just described.

And then the keynote just went to “eXperience”. It was wow, everything was about HXM or Human eXperience Management, not Human Capital Management. Look, I dislike the HCM idea of treating employees as “Capital”, that’s not what we try to do, but still there is a whole bunch more the the successful running of employees in a business than just looking after their experience!

I observed cynically to someone (the night club from night before possibly didn’t help my mood) that it seemed  that the SuccessFactors exec team had all been given bigger KPIs to sell Qualtrics than they had to sell SuccessFactors.

 

After the keynote were lots of general sessions including loads of roadmap sessions! The conference team really listened to the customer desire for roadmap sessions and most were repeated at least one. Even better the sessions were short on presentation and long on questions. This is why people come to conferences rather than reading the slides at home (and without having to fly stupid distances around the world). So SUPER KUDOS to the SuccessFactors team, your roadmap sessions were AWESOME! I had lots of fun in one run by Mark McCawley who I meet for the first time and leaked some news that I’m hoping we can reveal soon that will make many customers very happy!

Saw mention that instance sync tool will allow copy back of employee data from prod to non-prod systems (without full system refresh) allowing for trouble shooting of issues in test environments – this is awesome and will be a huge benefit for customers. When you look at the list price that Accenture were/are charging for a SAP CP tool that does similar thing, you can see how customers were really wanting this functionality!

Talking of the SAP App Store and SAP CP extensions, they were highly visible this year on the show floor and customers were clearly interested!

Final keynote was more eXperience Qualtrics puff along with 3 partner add on solutions, and a strange plug for SAP.io which I’m pretty sure the audience didn’t care about. Much disappointment that non of the partners on the stage with SuccessFactors add ons were using SAP Cloud Platform – a real lost opportunity to showcase the power and benefit of having extensions that look like they are just additional modules of SuccessFactors, a functionality that SuccessFactors has that is unique – I’m afraid integration to ServiceNow isn’t really that unique…

Last day and the team:

went to the golf driving range, where I discovered I’m not terrible at this golf lark, but not good at it either. Was fun!

Anyway – flight about to be called.

Was great to be at SuccessConnect, think it would be good for the SuccessFactors exec team to perhaps take a moment to listen to the SAP Mentor feedback – perhaps we can arrange something for next year?

rocks balanced on top of each other being splashed vigorously by a wave

Being “fair” vs being “balanced”

So, first up, I’m not intending to call anyone out here for not doing the right thing or even for doing the right thing – it’s just something happened recently that I feel I want to share and perhaps get some feedback about. That’s why I started writing here after all!

So, last year, my daughter decided that she’d put herself forward for part of the primary school technology leadership team. This group helps work with the school on stuff like spreading the messages about being safe online, how to use tech responsibly, teaching kids about cyber bullying and other really important things. I think they do some other things too, but with the limited budget the school has for tech, it limits what they can influence. I was proud that she was willing to put her hand up for what, even in our fairly affluent and well educated school, is a group which is still overwhelmingly male dominated.

Fast forward to this year (school years run to calendar years here in Australia, with summer holidays over end December to end of January.) She had been preparing with me a speech on why she would be an excellent choice to lead this group. Last Friday evening I found out that she had not been chosen as the lead for the group, but also had not had a chance to give her speech.

“Why?” I asked.

So the teacher had sat down with the technology leadership group, 8 boys and 2 girls. He asked them if they wanted to makes speeches to decided who should be appointed to lead the team (two leaders). The majority didn’t want to do speeches, so they decided not to do them. So then he asked them if they thought that it would be a good idea to have one girl and one boy as leaders. The majority didn’t think this fair given only two girls in the group. The teacher then went back to the girls and asked them what they thought. Unsurprisingly, the girls did not deviate from the majority position. So then they went around the group and each person nominated another person for the leadership role, explaining why they thought that person would be a good leader. The two people with most nominations were then “elected” as the leaders of the group. Two boys as the official leads of the technology leadership team.

Now, I took exception to this process.

Firstly, it’s not right to ask an unrepresentative group to elect leaders/representatives on behalf of a much larger group. In this case a heavily male dominated group made decisions about who should be leading and representing for the whole school population (gender ratio about 50/50). We can see what such a process has done to our political parties – but that’s another conversation…

Secondly, it’s not fair to ask a group of girls that are already in the minority to go against the wishes of their peers, especially when those wishes have just been publicly voiced.

Thirdly (and perhaps most controversially) this shouldn’t have been a choice made by the students at all.

I am very much aware that there is a real lack of upcoming talent in the IT industry here in Australia, and even more so a lack of female talent. I make special efforts to recruit those females that I see working in IT into my team. (If you’re a female in Melbourne Australia, working with cloud tech and interested in SAP cloud technology and building cool stuff, send me a CV, I will very likely find an opening for you!) My experience has been that given all the hassle most women have gone through to get into a tech career that they are far more capable and resilient than their male counterparts (sorry guys, but you just don’t have to work as hard as the girls do to get the same level of respect.)

I feel that we need to promote tech at the earliest possible stage of schooling and we need to make sure that it’s not seen as “something for the boys”. Given the unrepresentative numbers in the group, it looks like this hasn’t been happening well enough at my local school. So we need to give it a push. It needed to be shown to the younger girls that tech was something that girls could do, they needed their own leader, it should have been a girl.

So, I’ve had a conversation about this with the teacher concerned and in hindsight he can absolutely see the problems and the issues. He’s going to try to figure out what can be done. (He comes across to me as an excellent teacher who perhaps wasn’t considering the gender issues as much as I think we both think he might have done. I’m very glad that someone as willing to take feedback as he is teaching my child. I feel very lucky.) I’d hope that we end up with three leaders in the group at least one of them female – will be hard to take away that “elected” role from one of the boys. But closing the stable door when the horse is out already is hard.

So, I’ll wait and see what happens next. My daughter is giving me the cold shoulder for being “mean” to her favourite teacher, but I just couldn’t stand back and act like this didn’t involve me.

Bringing “balance” to an industry so heavily dominated by males will mean that sometimes we will have to make decisions that many will regard as unfair. Diversity means so much more than just gender diversity, but it’s probably one of the most obvious areas that we can address and should actively address. I’m loving that SuccessFactors is bringing “business beyond bias” to the workplace. But I think we need to proactively address this well before we hit the workplace. Wouldn’t it be cool if the same data analysis tools that we’re deploying into our work could also be used in our schools to help pick up bias and remove it?

Ok – that’s all, if you have any comments, please hit me up on twitter – @wombling would love to hear if you think there should have been a different way I could have handled this, or any suggestions on how to help promote WIT at the local school.

Cheers!

 

 

 

Link

Screenshot from https://www.workday.com/en-us/applications/workday-cloud-platform.html

So Jarret Pazahanick, posted me a link to the details of the more detailed announcement of the Workday PaaS (it having been earlier hinted at https://diginomica.com/2017/07/11/workday-finally-pops-paas/  http://blogs.workday.com/open-up-workday-cloud-platform/ a couple of months back.

He wondered at my thoughts. Well, here they are.

Firstly, I’m really glad Workday are saying they are releasing a PaaS, it justifies all the work that I have been doing in this space. Having a PaaS to support your SaaS based HR solutions is now pretty much de-facto table stakes.

On the downside, Workday didn’t actually announce a PaaS at all. They have suggested that developers use other PaaS to deploy and run the applications and then integrate the applications via a newly released suite of APIs.

I think the most telling thing about Workday’s PaaS is the disclaimer on their website. It isn’t publicly available and they may not deliver it at all.

Whilst the claim on the website is “The Workday Cloud Platform is built on the principle of openness” it’s not possible to access the details of the developer website without requesting a user id, so I’m not even sure of the scope of the APIs that they are releasing.

In fairness this is a similar situation to SuccessFactors about 5 to 6 years ago. Then even high level details of their APIs were a guarded secret. I remember one of my very first presentations about developing applications on what was then the SAP Netweaver Cloud, integrating to SuccessFactors Performance and Goals. It was painful to figure out how to do it. Now there are Open SAP MOOCs that step you through easy platform enabled steps.

https://open.sap.com/courses/cp3sf

Screenshot from Open SAP open.sap.com

I’m pretty sure that Workday will move to a more open model, the industry demands it, but until they work out a way to deliver an actual platform that their customers can run apps on, not just the services to support the apps, then I think we should really call them to task for calling something that is NOT a PaaS, a PaaS.

It’s nice that Workday feel they need to have a PaaS, but what I’ve seen doesn’t count as that, at least not in my understanding of the term.

 

On ABAP in the Cloud

ABAP in the cloud

Michael Koch kicked it all off with a tweet,

to which of course I had to reply:

then I was prodded:

and prodded:

and then James beat me to the blog:

and if you haven’t read James’ post, please do, it is excellent.

So whilst I’m waiting to hear how much it’s going to cost me to fix my car of which the engine has decided to stop working whilst on the way to work today, I thought that rather than drinking a bottle of Pinot Gris and attempting to forget about the shitty waste of a day I’ve had, I’d do something useful, productive (this post), and drink beetroot, apple, ginger and celery juice instead.

So here are thoughts upon which I will rant.

  • ABAP is a proprietary language which make its code costly to support.
  • Building for cloud is far more than just supporting cloud systems.
  • If you love ABAP to the exclusion of everything else that’s your bed, you lay in it. I like beetroot juice, I am so going to have pink pee later.
  • Java is the boring enterprise language of choice.
  • A PaaS really should be language agnostic, if not it’s a pretty crappy PaaS.
  • Why on earth have we ended up here? Who is paying for this?
  • Evolve or die.

These are all going to get intermixed in this rant, but I will still try to address them one by one.

Firstly, on the joys of ABAPers. I have discussed and even written about this, and it may just be the particular markets where I play, but it’s damn hard to find a good and excited ABAPer. People don’t learn the language unless they want to work on SAP products. Imagine how quickly that strips out the fun people. But where people have got good ABAP skills, they tend to have far more than that, also great business process understanding (Robbo has recently written about this https://blogs.sap.com/2017/10/02/abap-in-sap-cloud-platform-why/ ) Have a read, especially if you fall into the ABAP diehards camp, it will make you feel much happier than this blog post will.

But because the good ABAP folk have such great depth of business process understanding, they command a reasonable rate – and why not having a BA and a coder in one is a bit of a win is it not? So they are expensive. One hopes because they deliver better, but I find this is not true. They just cost more. But you have to have them to support the huge monolith that is your SAP ERP system. So embedded in companies around the world are these folk who can code ABAP, understand their systems and are if not well paid, expensive to have hanging around.

And you won’t find someone off the street who has just learnt ABAP who is useful, because the skill in ABAP isn’t in the language, it’s in understanding the existing library of  standard code and frameworks that you can use to get things done.

FFS the language still doesn’t have the concept of a Boolean!

The requirement for ABAP support is one of the reasons that SAP costs a decent amount to run. In the future as we move to S/4HANA public cloud (and we will, slowly but inevitably) cost saving will be essential. ABAP costs, so get rid of it in the equation. Out-source your custom development, even better, purchase it as SaaS from someone else, are you a custom software development house? No – they why do you try to build your own software? Concentrate on dishwasher powder, chocolate bars, beer or whatever it is you have as your core.

If we start building cloud extensions in ABAP we are locking down the list of people who could support them. This will cost us extra. Having worked with SaaS for the last few years, I can clearly state, cost of delivery is far more important now than it ever was on-prem. The expectations of customers are different. They will not pay the same amount to build an extension as they paid for the SaaS solution it enhances. ABAP ain’t cheap, and neither are ABAPers.

I don’t think ABAP and it’s whole lifecycle management is really well designed to build cloud apps. James mentioned some great points in his blog around dependency management, and how ABAP doesn’t support non-linear and project based development (hopefully ABAPGit will help here, the official voice of support from SAP is very encouraging.) But having spent the last 5 years build cloud apps that integrate to SAP systems, I have been so impressed by the huge amount of standard tooling and functionality that is available for projects outside of SAP. Like have you used Maven? It’s fricking awesome! To consider even thinking about managing the huge number of libraries that I use in most of my builds to do without this tooling would be unthinkable. Since James was probably more detailed and eloquent on this point I will stop there. But really, even if SAP support ABAPGit there is a hell of a long way to go to even think of being put into an imaginary cloud development language magic quadrant chart, let alone featuring anywhere but bottom left.

#ABAPisntDead. No of course it isn’t, there will be legacy on prem apps that will run and people will make businesses out of it, like those Rimini Street folk. But if you can’t see anything out there other than ABAP, my goodness you are short sighted. Any good programmer out there should be able to code in js (server side or browser), and should have a grasp of at least 2 other languages. If you can only deal with one, you’re not a programmer, you’re a liability for the people you work with. Having multiple skills is important, and it’s also important to know when to use them. Enlighten yourselves people, there is a whole world full of cool shite out there, go and have a look. If my post infuriates you because you believe that ABAP is the best thing ever, awesome, both for you and for me, because you have passion, go and use it, and me because it means I actually got some people who don’t agree with me to read this.

Java is boring, and safe, and commodity. And that is exactly what businesses love. You want something that is reliable, has been proven, does the job. Moreover, you want bucket loads of libraries that other people have built and tested that can do the things you want to do. Whilst I built an implementation of TFA that was compatible with Google’s TFA Authenticator app in ABAP, it was a pain in the arse, and hasn’t been updated since I wrote it and then worried about releasing it as open source because you weren’t allowed to do that with ABAP. There’s a standard lib for Java. Standard boring languages are the bedrock of good enterprise builds. I do like to play with server side js, (aka Node) but i’m still a sucker for strongly typed languages.

But if you don’t like Java, then awesome, choose something else. Indeed it should not matter what you choose, because any PaaS you build on should be language agnostic when it comes to providing services to you to consume. If you’re not consuming any services from your PaaS then you missed the memo about cloud development, please go back to your application server. A PaaS offers micro-services that should be able to be consumed by any application running on that platform. This inherently makes those services consumable in a fashion that is hard to use for ABAP and pretty standard for every other language. I’m sure that SAP could wrap their services into a consumable layer that would be easier to use in the Cloud based ABAP. But this then means we start losing one of the best bits of the PaaS, that it shouldn’t favour any runtime. We’ll see how this story plays out…

Which kinda segues into my next worry/rant/observation. How did we get here that a language that really isn’t suited to cloud extension ends up as an officially supported run time in SAP’s CF PaaS? This goes back to my original tweet.

I believe that it is clearly SAP’s strategy to move to the largest part of their revenue coming from public cloud based SaaS solutions (including ERP). Btw, I think this is a sound strategic vision, because if they don’t pivot to get there, someone else will take that space. The on-prem model will not make as much money in the future, todays small companies are tomorrows giants, and with SaaS solutions they don’t need to migrate/upscale, they will keep the solution they buy today. SAP needs to be in that space, and they need credibility that comes from large customers being there too.

To this end I envisage SAP have been discussing moving some very influential customers to the public cloud. Those customer, I would guess, have responded that they don’t want to loose their current people or custom build investments.

The obvious solution from SAP is to put together an ABAP cloud runtime. I cannot be cheap to do this though. The effort to make ABAP into a secure and lightweight containerizable solution will not be something that a team will do in a week or two. There must be some sound and solid business reasons to do this. For all the reasons I have previously mentioned I believe that if companies want to extend SAP SaaS solutions, they should think about using other languages, not ABAP. But I fear this is not about making a better solution, it is about making a marketable one. If customers believe that they can extend the value of their existing investments and also benefit from moving to SaaS based solution, that is a great sales pitch. It’s having your cake and eating it.

This vision (even if it doesn’t work out to be the reality) of a simple gateway to moving to SaaS ERP is what I believe we are now being sold. This isn’t a story for developers, this is a story for the high level execs that sign the S/4HANA subscriptions.

I hope that a cloud based ABAP will be the gateway that enables some organisations to get off the on-premise mode and head to the cloud. What I fully expect is that once they are there, they will realise that there are better and more supportable ways to extend. That would be great. In the meantime I fear that we start bringing non-cloudy ways of working into the cloud landscape, this will likely cause failed/cost overrun projects. We run the risk of preferring Cloud ABAP as a way to interact with S/4HANA cloud, that would be disastrous.

It has been suggested that Cloud ABAP will potentially be the solution that encourages adoption of the SAP Cloud Platform. I just hope it isn’t the solution that kills it. I would much rather the money being spent of putting ABAP into the cloud is used to handle some of the other issues I see with SAP CP, but clearly there is a view that it will be a return on investment.

Then again, if you’re not trying new stuff and making mistakes, you’re not learning. If you’re not learning, you’re falling behind. So here’s to making mistakes and learning! To steal the excellent closing lines from James’ post:

So buckle up because there’s no turning back at this point. It’s either evolve or die.

I look forward to a lively debate on this topic.

(James Wood – https://blogs.sap.com/2017/10/04/abap-in-the-cloud-is-this-a-good-thing/)

James, I couldn’t say it better mate. Although I would refer to the platform as SAP CP 😉

I think SAP Cloud Platform is and will be a key part of the story of SAP’s  and customers’ evolution to the cloud. If it takes putting an “runs ABAP” badge on it, to get people to see how useful it is, I’ll deal with it. But for sure, it would not be my recommendation to any organisation that it would be best practice. I’ll keep an open mind, perhaps it will be one day, if so I’ll adapt and evolve – because that’s what you should do.

As always, my own thoughts, not my company’s,  please feel free to jump onto SCN and reply to James’ post. I’ll probably read those comments as well as whatever gets posted on twitter.

 

It’s been a long time

Wow…

so like I haven’t posted anything for 2 years?

Hard to believe. But so easy to believe.

I’ve been busy. To say the least. I’ve built out multiple SAP SuccessFactors Extensions, expanded my team and worked some very long hours.

I haven’t stopped presenting or putting forward ideas. But I did stop writing about them.

Which is silly.

Let me see, I’m sure there is something that I can write about…

Yes, HTTPS – you’ll note that the site now is HTTPS. Powered by Let’s Encrypt. FREE!

That is very cool.

Perhaps I’ll write next about how I’ve done exactly the same thing to my SAP demo instance so it uses free let’s encrypt certs to enable HTTPS.

But perhaps tomorrow. 🙂

baby steps.