Config Sets for Custom Indexes Search Stax

After working with Sitecore Managed Cloud for almost two plus years, my first encounter with managed cloud was in the year 2020. I blogged about all my learnings on Managed Cloud in series of blog posts. After all this time, I still bump in to something new with different implementations across various clients.

For instance, this client had custom indexes. I always frown about custom indexes because I personally think they are only needed in some very specific scenarios and there are bunch of blog posts out there to help you understand when you would actually need them. In this specific Sitecore instance, they need them because.. lol I dont know. We inherited this solution, so, that is that. Now, these custom indexes will not be magically set up by Sitecore when you ask them to create the environments. So, how do you go about it? Here are the steps below.

  1. SOLR credentials are provided to you or your team on the support ticket you may have created for Sitecore to whip up new environment
  2. Log on to that admin console of SOLR
  3. From there – you can create the custom collections you like – Name them with preferably best practices around it
  4. Also, of course your index configurations should be patched well with all that sync strategies per your needs
  5. Now, go to Control Panel and rebuild your custom index – Boom – Yes ? No…wait..but

There is always buts right? Okay, my custom index has custom config set and not the one Sitecore may have created by default when they create your instance. 🙂

So, how do we create new collection on SOLR console using custom schema? It is actually quite simple with some twists. Like I always say, I will not repeat myself on the stuff that is already out there. I followed below references to actually understand the HOW part. Here it goes:

https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0848433

https://www.searchstax.com/docs/searchstax-cloud-dedicated-deployment-quick-start/#zkcli

Magic sauce is actually in the second link (section: Upload a SOLR configuration), but, first link is also important because that tells you exactly what options you have and what to do depending on option you picked. I picked option #2 noted on there which is to use zkcli. Yeah yeah, I am lazy you may think, but, I would like to think I am strategic. 😉

In my case option #2 will suffice as I have very simple needs and my managed schema config sets are actually checked in on source control, so, that is better approach than doing gymnastics with API. Most of the steps are clear in the second link above, but, I found their was additional ‘/’ that was throwing off my command. Plus, reminder if you are going with this option, you have to ask Sitecore via support ticket to whitelist your IP Range or IP Address. If you do not do this, command will not work and throw time out exceptions.

Before you run the below steps download zkcli https://github.com/searchstax/searchstax-client/archive/master.zip – Place the extracted version in your drive some where. It does not really matter where as long as you can go there.

I used powershell and I did CD to go the scripts folder inside the zkcli download mentioned above. Run the command that looks like below:

.\zkcli.bat -zkhost {zookeeper ensemble entry I grabbed from support ticket of environment creation details} -cmd upconfig -confdir ../configsets/{custom foldername inside your zkcli download }/conf/ -confname {name for config that would appear in SOLR}

Replace {with stuff in here} -> actual stuff noted (do not use those braces, I used them to suggest it is a placeholder and not real value)

If your IP is whitelisted and if you have JAVA installed with proper PATH variable set and yes if you are lucky, you will see your shiny new config set on SOLR once the command runs successfully. Once you have the config sets available, go back to your basic steps and when you are creating the collection you get to pick the custom config set.

Have fun you all!! I am sure more search stax interactions are yet to come soon. In the mean time, have fun.

Experience profile search not working

Do you remember an issue that took almost a month to resolve which constantly was in the back of you head bothering you and reminding you that it is still pending? This case is one such incident.

Here it goes the whole deal ! I appreciate Sitecore support team’s resilience and how they had been proactive in responding. But, I also feel that this should be like known or documented because guess what who ever used xDB before knows there could be challenges and they have to solved quickly to mitigate data latency issues.

Problem: Experience Editor search was not working on older Contacts as PII indexing was not enabled and later enabled.

Story: Human error, it happens even after you have documented painfully every single step that need to be done manually after a deployment. Unfortunately config edits on Processing server or Search Indexer side of things are manual deployments on our end at this time. Anything we push to CM/CD roles is absolutely one click deployments using patches as a standard. So, I missed doing this step noted here during production deployment. I realized the problem noted above could be because of that and changed it to ‘true’ on applicable roles/locations on Managed Cloud and hoped for the best. When I did this search did start to work, but, was only working on newer Contacts that got added to xDB post the change I had made. Why?

Resolution: I had this issue before when I first started playing with xDB, Experience profile and various other xConnect related services. I was pretty confident I need to rebuild xDB Index that is usually answer for such funky behaviors. I did that by doing the below steps:

  • Open up Azure portal and open up resource group of concern
  • Go to specific resource for xc-search app service
  • Open up Advanced Tools and hit Go -> this will take you to Kudu
  • Selected Powershell option -> Drilled down all the way in to Index Worker location and followed steps noted here
  • It said the rebuild succeeded, but, nope issue is not resolved and still search does not work on older contacts.

I was not sure on what else to do, I logged a support ticket with Sitecore and tried explaining what I tried so far. They confirmed that rebuild is what I should be doing in order to resolve the issue on hand. Strange, but, I did that exactly what is written on documentation. It turns out, rebuild has to be run subtly differently and in different location when the matter is about Managed Cloud. Support suggested that I do the rebuild command on location below such as below –

D:\local\Temp\jobs\continuous\IndexWorker\<Random Web Job Name>

Command is also subtly different from documentation it should be .\Sitecore.XConnectSearchIndexer.exe -rr

And then I started patiently monitoring the rebuild process using instructions here https://doc.sitecore.com/developers/100/sitecore-experience-platform/en/monitor-the-index-rebuild-process.html

You can also monitor the same from SOLR following below steps that were shared by Sitecore support :

  • Go to the admin panel
  • Switch to the xdb collection and click “query”
  • Then run the query: id:”xdb-rebuild-status”
  • This will tell us the exact current rebuild status of your xdb index.

Yep, I did all of that and every single time I tried it was stuck at 95% in finishing state, it never processed completely. So, Sitecore support asked me to do lot of steps to debug further to enhance the logging and log more things to index worker logs to help us understand why it is stuck and not completing. They identified the issue to be a setting that is higher than default one. The funny thing is, we did not set these settings up, they came with Managed Cloud. Any way below is the setting that I had to swap

Go to location below on Kudu: “D:\home\site\wwwroot\App_Data\jobs\continuous\IndexWorker\App_Data\config\sitecore\SearchIndexer\sc.Xdb.Collection.IndexWriter.SOLR.xml” )

I could see the config below:
<Solr.SolrWriterSettings>
<Type>Sitecore.Xdb.Collection.Search.Solr.SolrWriterSettings, Sitecore.Xdb.Collection.Search.Solr</Type>
<LifeTime>Singleton</LifeTime>
<Options>
<ConnectionStringName>solrCore</ConnectionStringName>
<RequireHttps>true</RequireHttps>
<MaximumUpdateBatchSize>1000</MaximumUpdateBatchSize>
<MaximumDeleteBatchSize>1000</MaximumDeleteBatchSize>
<MaximumCommitMilliseconds>600000</MaximumCommitMilliseconds>
<ParallelizationDegree>4</ParallelizationDegree>
<MaximumRetryDelayMilliseconds>5000</MaximumRetryDelayMilliseconds>
<RetryCount>5</RetryCount>
<Encoding>utf-8</Encoding>
</Options>
</Solr.SolrWriterSettings>

“MaximumCommitMilliseconds” value was too high and it was recommended to change it to “1000” . This apparently was the default value. Suspicion is when best practices were followed according to KB article by Managed Cloud team they must have swapped it as part of default set up steps.

I did the above change and queued rebuild process again for nth time and monitored patiently. Worked!!! I almost cried out loud given I had to do these steps so many steps, waited so many days to get the issue resolved.

Hoping Sitecore team can update documentation around this specifically for Managed cloud, until then, hope this helps some one else loosing sleep over similar problem and needs to queue in that rebuild index.

Sitecore Tagging Surprises

Sitecore tagging is immensely powerful out of the box starting Sitecore 7+.  You can tag any items( including bucketed items)
The below links will get you started on the basic features –

https://visionsincode.wordpress.com/2014/11/16/tagging-is-fun-in-sitecore/

https://community.sitecore.net/technical_blogs/b/sitecore_7_development_team/posts/content-tagging-with-sitecore-7

https://doc.sitecore.net/sitecore_experience_platform/content_authoring/managing_items/general/add_a_tag_to_an_item

Now, coming to the surprises…

  1. While working with the basics,  the __Semantics field on my local sitecore instance did not look anything like the articles explain.  After some pondering,  it turns out my ‘raw’ view was checked.  So, for the multi-value search field to behave and allow selection as expected make sure this is unchecked.
  2. After adding some new tags into tag repository(),  when i go back to tagging section of my item, I did not see the newly added tags in the selection list.   This is because tagging depends and reads from indexes.  when i re-built my indexes, I could then see the new one’s
  3. Since, tagging is so dependent on indexing,  selecting a good indexing strategy is very important.  Doing this operation only as often as needed and not overdoing goes long way.
To tackle #3,  it would be handy to know the default indexing strategies on master and web databases.  For master, the default is syncMaster and for web database, the default strategy is onPublishEndAsync.   You can apply different one’s as needed, but, in our case the defaults would just do it. 
You can read more on indexing strategies available through Sitecore OOTB on the below article, Good reads. 🙂

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/index_update_strategies

https://community.sitecore.net/technical_blogs/b/sitecorejohn_blog/posts/sitecore-7-index-update-strategies

https://doc.sitecore.net/sitecore_experience_platform/setting_up__maintaining/search_and_indexing/indexing/rebuild_search_indexes