Finding Near-Optimal Configurations in Colossal Spaces with Statistical Guarantees

  1. Oh, Jeho 1
  2. Batory, Don 1
  3. Heradio, Rubén 2
  1. 1 The University of Texas at Austin, USA
  2. 2 Universidad Nacional de Educación a Distancia, Spain
Revista:
ACM Transactions on Software Engineering and Methodology

ISSN: 1049-331X,  1557-7392

Año de publicación: 2023

Tipo: Artículo

DOI: 10.1145/3611663 GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: ACM Transactions on Software Engineering and Methodology

Referencias bibliográficas

  • 2021. Learning software configuration spaces: A systematic literature review. JSS 182(2021). 2021. Learning software configuration spaces: A systematic literature review. JSS 182(2021).
  • 7z Website 2021. 7z Website. https://www.7-zip.org/download.html. 7z Website 2021. 7z Website. https://www.7-zip.org/download.html.
  • M. Acher et al. 2022. Feature Subset Selection for Learning Huge Configuration Spaces: the Case of Linux Kernel Size. In SPLC. M. Acher et al. 2022. Feature Subset Selection for Learning Huge Configuration Spaces: the Case of Linux Kernel Size. In SPLC.
  • D. Achlioptas Z.S. Hammoudeh and P. Theodoropoulos. 2018. Fast Sampling of Perfectly Uniform Satisfying Assignments. In SAT. D. Achlioptas Z.S. Hammoudeh and P. Theodoropoulos. 2018. Fast Sampling of Perfectly Uniform Satisfying Assignments. In SAT.
  • S. Agrawal S. Chaudhuri and V. Narasayya. 2000. Automated Selection of Materialized Views and Indexes in SQL Databases. In VLDB. S. Agrawal S. Chaudhuri and V. Narasayya. 2000. Automated Selection of Materialized Views and Indexes in SQL Databases. In VLDB.
  • D. Van Aken A. Pavlo G.J. Gordon and B. Zhang. 2017. Automatic Database Management System Tuning Through Large-Scale Machine Learning. In SIGMOD. D. Van Aken A. Pavlo G.J. Gordon and B. Zhang. 2017. Automatic Database Management System Tuning Through Large-Scale Machine Learning. In SIGMOD.
  • Apache Web Server 2002. Apache HTTP Server Project. https://httpd.apache.org/. Apache Web Server 2002. Apache HTTP Server Project. https://httpd.apache.org/.
  • S. Apel D. Batory C. Kästner and G. Saake. 2013. Feature-Oriented Software Product Lines. Springer. S. Apel D. Batory C. Kästner and G. Saake. 2013. Feature-Oriented Software Product Lines. Springer.
  • B.C. Arnold N. Balakrisnhan and H.N. Nagaraja. 2008. A First Course in Order Statistics. SIAM. B.C. Arnold N. Balakrisnhan and H.N. Nagaraja. 2008. A First Course in Order Statistics. SIAM.
  • axTLS 2018. AxTLS Website. http://axtls.sourceforge.net/. axTLS 2018. AxTLS Website. http://axtls.sourceforge.net/.
  • R.A. Aziz G. Chu C.J. Muise and P.J. Stuckey. 2015. #∃SAT: Projected Model Counting. In SAT. R.A. Aziz G. Chu C.J. Muise and P.J. Stuckey. 2015. #∃SAT: Projected Model Counting. In SAT.
  • D. Batory. 2005. Feature Models Grammars and Propositional Formulas. In SPLC. D. Batory. 2005. Feature Models Grammars and Propositional Formulas. In SPLC.
  • D. Batory . 2021. Automated Software Design : Volume 1 . Lulu Press . D. Batory. 2021. Automated Software Design: Volume 1. Lulu Press.
  • D. Batory and C. Gotlieb. 1982. A Unifying Model of Physical Databases. ACM TODS (Dec. 1982). D. Batory and C. Gotlieb. 1982. A Unifying Model of Physical Databases. ACM TODS (Dec. 1982).
  • D. Batory J. Oh R. Heradio and D. Benavides. 2021. Product Optimization in Stepwise Design. In Logic Computation and Rigorous Methods A. Raschke E. Riccobene and K.D. Schewe (Eds.). Springer. D. Batory J. Oh R. Heradio and D. Benavides. 2021. Product Optimization in Stepwise Design. In Logic Computation and Rigorous Methods A. Raschke E. Riccobene and K.D. Schewe (Eds.). Springer.
  • BDDSampler 2022. BDDSampler Website. https://github.com/davidfa71/BDDSampler. BDDSampler 2022. BDDSampler Website. https://github.com/davidfa71/BDDSampler.
  • J. Bergstra and Y. Bengio. 2012. Random Search for Hyper-Parameter Optimization. JMLR (2012). J. Bergstra and Y. Bengio. 2012. Random Search for Hyper-Parameter Optimization. JMLR (2012).
  • BerkeleyDB Website 2021. BerkeleyDB Website. https://www.oracle.com/database/technologies/related/berkeleydb.html. BerkeleyDB Website 2021. BerkeleyDB Website. https://www.oracle.com/database/technologies/related/berkeleydb.html.
  • A. Biere M. Heule H. Maaren and T. Walsh. 2009. Handbook of Satisfiability: Volume 185 Frontiers in Artificial Intelligence and Applications. IOS Press. A. Biere M. Heule H. Maaren and T. Walsh. 2009. Handbook of Satisfiability: Volume 185 Frontiers in Artificial Intelligence and Applications. IOS Press.
  • A. Bluman . 2009. Elementary Statistics: A Step by Step Approach . McGraw-Hill Higher Education . A. Bluman. 2009. Elementary Statistics: A Step by Step Approach. McGraw-Hill Higher Education.
  • L. Breiman J. Friedman R. Olshen and C. Stone. 1984. Classification and Regression Trees. Wadsworth and Brooks/Cole Advanced Books and Software. L. Breiman J. Friedman R. Olshen and C. Stone. 1984. Classification and Regression Trees. Wadsworth and Brooks/Cole Advanced Books and Software.
  • P. Bruce A. Bruce and P. Gedeck. 2020. Practical Statistics. O’Reilly. P. Bruce A. Bruce and P. Gedeck. 2020. Practical Statistics. O’Reilly.
  • R. Bryant et al. 2007. Deciding Bit-vector Arithmetic with Abstraction. In TACAS. R. Bryant et al. 2007. Deciding Bit-vector Arithmetic with Abstraction. In TACAS.
  • S. Chakraborty D.J. Fremont K.S. Meel S.A. Seshia and M.Y. Vardi. 2015. On Parallel Scalable Uniform SAT Witness Generation. In TACAS. S. Chakraborty D.J. Fremont K.S. Meel S.A. Seshia and M.Y. Vardi. 2015. On Parallel Scalable Uniform SAT Witness Generation. In TACAS.
  • S. Chakraborty K.S. Meel and M.Y. Vardi. 2013. A Scalable and Nearly Uniform Generator of SAT Witnesses. In CAV. S. Chakraborty K.S. Meel and M.Y. Vardi. 2013. A Scalable and Nearly Uniform Generator of SAT Witnesses. In CAV.
  • S. Chakraborty K. Meel and M. Vardi. 2013. A Scalable Approximate Model Counter. In CP. S. Chakraborty K. Meel and M. Vardi. 2013. A Scalable Approximate Model Counter. In CP.
  • S. Chakraborty K.S. Meel and M.Y. Vardi. 2014. Balancing Scalability and Uniformity in SAT Witness Generator. In DAC. S. Chakraborty K.S. Meel and M.Y. Vardi. 2014. Balancing Scalability and Uniformity in SAT Witness Generator. In DAC.
  • S. Chaudhuri. 1998. An Overview of Query Optimization in Relational Systems. In PODS. S. Chaudhuri. 1998. An Overview of Query Optimization in Relational Systems. In PODS.
  • T.Y. Chen H. Leung and I.K. Mak. 2004. Adaptive Random Testing. In ASIAN. T.Y. Chen H. Leung and I.K. Mak. 2004. Adaptive Random Testing. In ASIAN.
  • CUDD Website 2022. CUDD Website. https://github.com/vscosta/cudd. CUDD Website 2022. CUDD Website. https://github.com/vscosta/cudd.
  • K. Deb A. Pratap S. Agarwal and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE TEVC (April 2002). K. Deb A. Pratap S. Agarwal and T. Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE TEVC (April 2002).
  • J. Dorn S. Apel and N. Siegmund. 2020. Mastering Uncertainty in Performance Estimations of Configurable Software Systems. In ASE. J. Dorn S. Apel and N. Siegmund. 2020. Mastering Uncertainty in Performance Estimations of Configurable Software Systems. In ASE.
  • R. Dutra K. Laeufer J. Bachrach and K. Sen. 2018. Efficient Sampling of SAT Solutions for Testing. In ICSE. R. Dutra K. Laeufer J. Bachrach and K. Sen. 2018. Efficient Sampling of SAT Solutions for Testing. In ICSE.
  • D. Fernandez-Amoros , S. Bra , E. Aranda-Escolastico , and R. Heradio . 2020 . Using Extended Logical Primitives for Efficient BDD Building. https://www.mdpi.com/2227-7390/8/8/1253 . Mathematics 8 , 8 (2020), 1 – 17 . D. Fernandez-Amoros, S. Bra, E. Aranda-Escolastico, and R. Heradio. 2020. Using Extended Logical Primitives for Efficient BDD Building. https://www.mdpi.com/2227-7390/8/8/1253. Mathematics 8, 8 (2020), 1–17.
  • D. Fernandez-Amoros R. Heradio C. Mayr-Dorn and A. Egyed. 2019. A Kconfig translation to logic with one-way validation system. In SPLC. D. Fernandez-Amoros R. Heradio C. Mayr-Dorn and A. Egyed. 2019. A Kconfig translation to logic with one-way validation system. In SPLC.
  • D. Fernandez-Amoros R. Heradio C. Mayr-Dorn and A. Egyed. 2022. Scalable Sampling of Highly-Configurable Systems: Generating Random Instances of the Linux Kernel. In ASE. D. Fernandez-Amoros R. Heradio C. Mayr-Dorn and A. Egyed. 2022. Scalable Sampling of Highly-Configurable Systems: Generating Random Instances of the Linux Kernel. In ASE.
  • A. Field , J. Miles , and Zoë Field . 2012. Discovering Statistics Using R . SAGE Publications LTD. A. Field, J. Miles, and Zoë Field. 2012. Discovering Statistics Using R. SAGE Publications LTD.
  • P. Gazillo , J. Oh , and M. Myers . 2019 . Uniform Sampling from Kconfig Feature Models . Technical Report TR-19-02. University of Texas at Austin, Department of Computer Science. P. Gazillo, J. Oh, and M. Myers. 2019. Uniform Sampling from Kconfig Feature Models. Technical Report TR-19-02. University of Texas at Austin, Department of Computer Science.
  • V. Gogate and R. Dechter. 2006. A New Algorithm for Sampling CSP Solutions Uniformly At Random. In CP. V. Gogate and R. Dechter. 2006. A New Algorithm for Sampling CSP Solutions Uniformly At Random. In CP.
  • J. Guo et  al . 2017. SMTIBEA: a Hybrid Multi-objective Optimization Algorithm for Configuring Large Constrained Software Product Lines. SoSyM ( 2017 ), 1–20. J. Guo et al. 2017. SMTIBEA: a Hybrid Multi-objective Optimization Algorithm for Configuring Large Constrained Software Product Lines. SoSyM (2017), 1–20.
  • J. Guo et  al . 2018 . Data-efficient Performance Learning for Configurable Systems . Empirical Software Engineering 23 , 3 (2018), 1826 – 1867 . J. Guo et al. 2018. Data-efficient Performance Learning for Configurable Systems. Empirical Software Engineering 23, 3 (2018), 1826–1867.
  • J. Guo K. Czarnecki S. Apel N. Siegmund and A. Wasowski. 2013. Variability-aware Performance Prediction: A Statistical Learning Approach. In ASE. J. Guo K. Czarnecki S. Apel N. Siegmund and A. Wasowski. 2013. Variability-aware Performance Prediction: A Statistical Learning Approach. In ASE.
  • J. Guo and K. Shi. 2018. To Preserve Or Not to Preserve Invalid Solutions in Search-bassed Software Engineering: A Case Study in Software Produce Lines. In ICSE. J. Guo and K. Shi. 2018. To Preserve Or Not to Preserve Invalid Solutions in Search-bassed Software Engineering: A Case Study in Software Produce Lines. In ICSE.
  • H.264 Video Compression 2022. H.264 Video Compression. https://www.haivision.com/resources/streaming-video-definitions/h-264/. H.264 Video Compression 2022. H.264 Video Compression. https://www.haivision.com/resources/streaming-video-definitions/h-264/.
  • H. Ha and H. Zhang . 2019 . Deepperf: Performance Prediction for Configurable Software with Deep Sparse Neural Network. In ICSE. H. Ha and H. Zhang. 2019. Deepperf: Performance Prediction for Configurable Software with Deep Sparse Neural Network. In ICSE.
  • H. Ha and H. Zhang. 2019. Performance-Influence Model for Highly Configurable Software with Fourier Learning and Lasso Regression. In ICSME. H. Ha and H. Zhang. 2019. Performance-Influence Model for Highly Configurable Software with Fourier Learning and Lasso Regression. In ICSME.
  • A. Halin A. Nuttinck M. Acher X. Devroey G. Perrouin and B. Baudry. 2019. Test Them All Is It Worth It? Assessing Confguration Sampling on the JHipster Web Development Stack. Empirical Software Engineering(April 2019). A. Halin A. Nuttinck M. Acher X. Devroey G. Perrouin and B. Baudry. 2019. Test Them All Is It Worth It? Assessing Confguration Sampling on the JHipster Web Development Stack. Empirical Software Engineering(April 2019).
  • C. Henard M. Papadakis M. Harman and Y. Traon. 2015. Combining Multi-objective Search and Constraint Solving for Configuring Large Software Product Lines. In ICSE. C. Henard M. Papadakis M. Harman and Y. Traon. 2015. Combining Multi-objective Search and Constraint Solving for Configuring Large Software Product Lines. In ICSE.
  • R. Heradio et  al . 2022 . Uniform and Scalable Sampling of Highly Configurable Systems . Empirical Software Engineering 27 , 2 (2022), 44 . R. Heradio et al. 2022. Uniform and Scalable Sampling of Highly Configurable Systems. Empirical Software Engineering 27, 2 (2022), 44.
  • R. Heradio D. Fernandez-Amoros J.A. Galindo and D. Benavides. 2020. Uniform and Scalable SAT-Sampling for Configurable Systems. In SPLC). R. Heradio D. Fernandez-Amoros J.A. Galindo and D. Benavides. 2020. Uniform and Scalable SAT-Sampling for Configurable Systems. In SPLC).
  • M.J. Heule O. Kullmann S. Wieringa and A. Biere. 2011. Cube and Conquer: Guiding CDCL SAT Solvers by Lookaheads. In HVC. M.J. Heule O. Kullmann S. Wieringa and A. Biere. 2011. Cube and Conquer: Guiding CDCL SAT Solvers by Lookaheads. In HVC.
  • 10.1214/aoms/1177730150
  • H.H. Hoos. 2012. Automated Algorithm Configuration and Parameter Tuning. H.H. Hoos. 2012. Automated Algorithm Configuration and Parameter Tuning.
  • 10.1016/j.infsof.2017.10.018
  • F. Hutter H.H. Hoos and K. Leyton-Brown. 2011. Sequential Model-Based Optimization for General Algorithm Configuration. In LION. F. Hutter H.H. Hoos and K. Leyton-Brown. 2011. Sequential Model-Based Optimization for General Algorithm Configuration. In LION.
  • M.S. Iqbal , R. Krishna , M.A. Javidian , B. Ray , and P. Jamshidi . 2022 . Unicorn: Reasoning About Configurable System Performance Through the Lens of Causality. In EuroSys. M.S. Iqbal, R. Krishna, M.A. Javidian, B. Ray, and P. Jamshidi. 2022. Unicorn: Reasoning About Configurable System Performance Through the Lens of Causality. In EuroSys.
  • P. Jackson and D. Sheridan. 2004. Clause Form Conversions for Boolean Circuits. In SAT. P. Jackson and D. Sheridan. 2004. Clause Form Conversions for Boolean Circuits. In SAT.
  • P. Jamshidi et al. 2017. Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis. In ASE. P. Jamshidi et al. 2017. Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis. In ASE.
  • P. Jamshidi and G. Casale. 2016. An Uncertainty-aware Approach to Optimal Configuration of Stream Processing Systems. In MASCOTS. P. Jamshidi and G. Casale. 2016. An Uncertainty-aware Approach to Optimal Configuration of Stream Processing Systems. In MASCOTS.
  • M. Järvisalo A. Biere and M.J.H. Heule. 2010. Blocked Clause Elimination. In TACAS. M. Järvisalo A. Biere and M.J.H. Heule. 2010. Blocked Clause Elimination. In TACAS.
  • C. Kaltenecker A. Grebhahn N. Siegmund J. Guo and S. Apel. 2019. Distance-Based Sampling of Software Configuration Spaces. In ICSE. C. Kaltenecker A. Grebhahn N. Siegmund J. Guo and S. Apel. 2019. Distance-Based Sampling of Software Configuration Spaces. In ICSE.
  • M. Karnaugh . 1953. The Map Method for Synthesis of Combinational Logic Circuits . IEEE Communication and Electronics 72, 5 ( 1953 ). M. Karnaugh. 1953. The Map Method for Synthesis of Combinational Logic Circuits. IEEE Communication and Electronics 72, 5 (1953).
  • Kconfig Language 2018. Kconfig Language Specification. https://www.kernel.org/doc/Documentation/kbuild/kconfig-language.txt. Kconfig Language 2018. Kconfig Language Specification. https://www.kernel.org/doc/Documentation/kbuild/kconfig-language.txt.
  • Kconfig Tool 2018. Kconfig Tool Specification. https://www.kernel.org/doc/Documentation/kbuild/kconfig.txt. Kconfig Tool 2018. Kconfig Tool Specification. https://www.kernel.org/doc/Documentation/kbuild/kconfig.txt.
  • D.E. Knuth . 2009. The Art of Computer Programming , Volume 4 , Fascicle 1: Bitwise Tricks & Techniques, Binary Decision Diagrams. Addison-Wesley Professional . D.E. Knuth. 2009. The Art of Computer Programming, Volume 4, Fascicle 1: Bitwise Tricks & Techniques, Binary Decision Diagrams. Addison-Wesley Professional.
  • 10.1007/s10270-018-0662-9
  • J. Krall , T. Menzies , and M. Davies . 2015 . Gale: Geometric Active Learning for Search-Based Software Engineering . IEEE TSE (Oct . 2015). J. Krall, T. Menzies, and M. Davies. 2015. Gale: Geometric Active Learning for Search-Based Software Engineering. IEEE TSE (Oct. 2015).
  • S. Krieter. 2019. Enabling Efficient Automated Configuration Generation and Management. In SPLC. S. Krieter. 2019. Enabling Efficient Automated Configuration Generation and Management. In SPLC.
  • 10.1080/01621459.1952.10483441
  • O. Kullmann . 1999. On a Generalization of Extended Resolution. Discrete Applied Mathematics 96-97 ( 1999 ), 149 – 176. O. Kullmann. 1999. On a Generalization of Extended Resolution. Discrete Applied Mathematics 96-97 (1999), 149 – 176.
  • H. Kuo , J. Chen , S. Mohan , and T. Xu . 2020 . Set the Configuration for the Heart of the Os: On the Practicality of Operating System Kernel Debloating . POMACS 4 , 1 (2020), 1 – 27 . H. Kuo, J. Chen, S. Mohan, and T. Xu. 2020. Set the Configuration for the Heart of the Os: On the Practicality of Operating System Kernel Debloating. POMACS 4, 1 (2020), 1–27.
  • B. Lantz . 2019. Machine Learning with R . Packt Publishing . B. Lantz. 2019. Machine Learning with R. Packt Publishing.
  • H. Levene . 1960. Robust tests for equality of variances . Stanford University Press . H. Levene. 1960. Robust tests for equality of variances. Stanford University Press.
  • LLVM 2020. The LLVM Compiler Infrastructure. https://llvm.org/. LLVM 2020. The LLVM Compiler Infrastructure. https://llvm.org/.
  • S. Gocht M. Soos and K.S. Meel. 2020. Tinted Detached and Lazy CNF-XOR solving and its Applications to Counting and Sampling. In CAV. S. Gocht M. Soos and K.S. Meel. 2020. Tinted Detached and Lazy CNF-XOR solving and its Applications to Counting and Sampling. In CAV.
  • Mann Whitney 2020. Mann-Whitney U-Test (Wilcoxon Rank Sum Test). https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_nonparametric/BS704_Nonparametric4.html. Mann Whitney 2020. Mann-Whitney U-Test (Wilcoxon Rank Sum Test). https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_nonparametric/BS704_Nonparametric4.html.
  • B. Marker D. Batory and R. Geijn. 2014. Understanding Performance Stairs: Elucidating Heuristics. In ASE. B. Marker D. Batory and R. Geijn. 2014. Understanding Performance Stairs: Elucidating Heuristics. In ASE.
  • H. Martin et  al . 2021. Transfer Learning Across Variants and Versions: The Case of Linux Kernel Size . IEEE TSE (Sept . 2021 ). H. Martin et al. 2021. Transfer Learning Across Variants and Versions: The Case of Linux Kernel Size. IEEE TSE (Sept. 2021).
  • H. Martin M. Acher J.A. Pereira and J. Jézéquel. 2021. A Comparison of Performance Specialization Learning for Configurable Systems. In SPLC. H. Martin M. Acher J.A. Pereira and J. Jézéquel. 2021. A Comparison of Performance Specialization Learning for Configurable Systems. In SPLC.
  • MathIsFun.com. 2019. Standard Deviation Formulas. https://www.mathsisfun.com/data/standard-deviation-formulas.html. MathIsFun.com. 2019. Standard Deviation Formulas. https://www.mathsisfun.com/data/standard-deviation-formulas.html.
  • F. Medeiros C. Kästner M. Ribeiro R. Gheyi and S. Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In ICSE. F. Medeiros C. Kästner M. Ribeiro R. Gheyi and S. Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In ICSE.
  • D. Munoz J. Oh M. Pinto L. Fuentes and D. Batory. 2019. Uniform Random Sampling Product Configurations of Feature Models That Have Numerical Features. In SPLC. D. Munoz J. Oh M. Pinto L. Fuentes and D. Batory. 2019. Uniform Random Sampling Product Configurations of Feature Models That Have Numerical Features. In SPLC.
  • D. Munoz , J. Oh , M. Pinto , L. Fuentes , and D. Batory . 2022 . Nemo: A Tool to Transform Feature Models with Numerical Features and Arithmetic Constraints. In ICSR. D. Munoz, J. Oh, M. Pinto, L. Fuentes, and D. Batory. 2022. Nemo: A Tool to Transform Feature Models with Numerical Features and Arithmetic Constraints. In ICSR.
  • D.J. Munoz , M. Pinto , and L. Fuentes . 2018 . Finding Correlations of Features Affecting Energy Consumption and Performance of Web Servers Using the HADAS Eco-assistant . Computing 100 , 11 (Nov. 2018), 1155–1173. D.J. Munoz, M. Pinto, and L. Fuentes. 2018. Finding Correlations of Features Affecting Energy Consumption and Performance of Web Servers Using the HADAS Eco-assistant. Computing 100, 11 (Nov. 2018), 1155–1173.
  • S. Muroga . 1979. Logic Design and Switching Theory . John Wiley & Sons, Inc. S. Muroga. 1979. Logic Design and Switching Theory. John Wiley & Sons, Inc.
  • V. Nair T. Menzies N. Siegmund and S. Apel. 2017. Using Bad Learners to Find Good Configurations. In FSE. V. Nair T. Menzies N. Siegmund and S. Apel. 2017. Using Bad Learners to Find Good Configurations. In FSE.
  • V. Nair Z. Yu T. Menzies N. Siegmund and S. Apel. 2020. Finding Faster Configurations Using FLASH. IEEE TSE (July 2020). V. Nair Z. Yu T. Menzies N. Siegmund and S. Apel. 2020. Finding Faster Configurations Using FLASH. IEEE TSE (July 2020).
  • A. Nöhrer and A. Egyed. 2013. C2O Configurator: a Tool for Guided Decision-making. In ASE. A. Nöhrer and A. Egyed. 2013. C2O Configurator: a Tool for Guided Decision-making. In ASE.
  • J. Oh. 2022. Finding Near-Optimal Configurations in Colossal Product Spaces of Highly Configurable Systems. Ph. D. Dissertation . University of Texas at Austin, Dept. of Computer Science . J. Oh. 2022. Finding Near-Optimal Configurations in Colossal Product Spaces of Highly Configurable Systems. Ph. D. Dissertation. University of Texas at Austin, Dept. of Computer Science.
  • J. Oh D. Batory M. Myers and N. Siegmund. 2017. Finding Near-optimal Configurations in Product Lines by Random Sampling. In FSE. J. Oh D. Batory M. Myers and N. Siegmund. 2017. Finding Near-optimal Configurations in Product Lines by Random Sampling. In FSE.
  • J. Oh , P. Gazzillo , D. Batory , M. Heule , and M. Myers . 2020 . Scalable Uniform Sampling for Real-World Software Product Lines . Technical Report TR-20-01. Dept. of Computer Science , University of Texas at Austin. J. Oh, P. Gazzillo, D. Batory, M. Heule, and M. Myers. 2020. Scalable Uniform Sampling for Real-World Software Product Lines. Technical Report TR-20-01. Dept. of Computer Science, University of Texas at Austin.
  • R. Olaechea D. Rayside J. Guo and K. Czarnecki. 2014. Comparison of Exact and Approximate Multi-objective Optimization for Software Product Lines. In ICSE. R. Olaechea D. Rayside J. Guo and K. Czarnecki. 2014. Comparison of Exact and Approximate Multi-objective Optimization for Software Product Lines. In ICSE.
  • J. Pereira M. Acher H. Martin and J.M. Jézéquel. 2020. Sampling Effect on Performance Prediction of Configurable Systems: A Case Study. In ICPE. J. Pereira M. Acher H. Martin and J.M. Jézéquel. 2020. Sampling Effect on Performance Prediction of Configurable Systems: A Case Study. In ICPE.
  • P. Perrotta. 2020. Programming Machine Learning: From Coding to Deep Learning. The Pragmatic Bookshelf. P. Perrotta. 2020. Programming Machine Learning: From Coding to Deep Learning. The Pragmatic Bookshelf.
  • 10.1016/S0747-7171(86)80028-1
  • Q. Plazar M. Acher G. Perrouin X. Devroey and M. Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In ICST. Q. Plazar M. Acher G. Perrouin X. Devroey and M. Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In ICST.
  • T. Puiu. 2021. What Does 5-sigma Mean in Science?https://www.zmescience.com/science/what-5-sigma-means-0423423/. T. Puiu. 2021. What Does 5-sigma Mean in Science?https://www.zmescience.com/science/what-5-sigma-means-0423423/.
  • J.P. Royston . 1982 . An Extension of Shapiro and Wilk’s W Test for Normality to Large Samples . Journal of the Royal Statistical Society 31 , 2 (1982), 115 – 124 . J.P. Royston. 1982. An Extension of Shapiro and Wilk’s W Test for Normality to Large Samples. Journal of the Royal Statistical Society 31, 2 (1982), 115–124.
  • 10.1016/j.scico.2018.12.002
  • A. Sarkar J. Guo N. Siegmund S. Apel and K. Czarnecki. 2015. Cost-efficient Sampling for Performance Prediction of Configurable Systems. In ASE. A. Sarkar J. Guo N. Siegmund S. Apel and K. Czarnecki. 2015. Cost-efficient Sampling for Performance Prediction of Configurable Systems. In ASE.
  • S. Sharma R. Gupta S. Roy and K.S. Meel. 2018. Knowledge Compilation Meets Uniform Sampling. In LPAR. S. Sharma R. Gupta S. Roy and K.S. Meel. 2018. Knowledge Compilation Meets Uniform Sampling. In LPAR.
  • K. Shi. 2017. Combining Evolutionary Algorithms with Constraint Solving for Configuration Optimization. In ICSME. K. Shi. 2017. Combining Evolutionary Algorithms with Constraint Solving for Configuration Optimization. In ICSME.
  • S. Siegel and J. Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. MacGraw Hill. S. Siegel and J. Castellan. 1988. Nonparametric Statistics for the Behavioral Sciences. MacGraw Hill.
  • N. Siegmund et al. 2012. Dataset for Siegmund2012. http://fosd.de/SPLConqueror. N. Siegmund et al. 2012. Dataset for Siegmund2012. http://fosd.de/SPLConqueror.
  • N. Siegmund et al. 2012. Predicting Performance Via Automated Feature-interaction Detection. In ICSE. N. Siegmund et al. 2012. Predicting Performance Via Automated Feature-interaction Detection. In ICSE.
  • N. Siegmund A. Grebhahn S. Apel and C. Kästner. 2015. Performance-Influence Models for Highly Configurable Systems. In FSE. N. Siegmund A. Grebhahn S. Apel and C. Kästner. 2015. Performance-Influence Models for Highly Configurable Systems. In FSE.
  • N. Siegmund M. Rosenmüller M. Kuhlemann C. Kästner and G. Saake. 2012. SPL Conqueror: Toward Optimization of Non-functional Properties in Software Product Lines. Software Quality Journal(Sept. 2012). N. Siegmund M. Rosenmüller M. Kuhlemann C. Kästner and G. Saake. 2012. SPL Conqueror: Toward Optimization of Non-functional Properties in Software Product Lines. Software Quality Journal(Sept. 2012).
  • N. Siegmund S. Sobernig and S. Apel. 2017. Attributed Variability Models: Outside the Comfort Zone. In FSE. N. Siegmund S. Sobernig and S. Apel. 2017. Attributed Variability Models: Outside the Comfort Zone. In FSE.
  • G.J. Székely , M.L. Rizzo , and N.K. Bakirov . 2007 . Measuring and Testing Dependence by Correlation of Distances . The Annals of Statistics , 35 , 6 (2007), 27692794 . G.J. Székely, M.L. Rizzo, and N.K. Bakirov. 2007. Measuring and Testing Dependence by Correlation of Distances. The Annals of Statistics, 35, 6 (2007), 27692794.
  • M. Thurley. 2006. SharpSAT–counting Models with Advanced Component Caching and Implicit BCP. In SAT. M. Thurley. 2006. SharpSAT–counting Models with Advanced Component Caching and Implicit BCP. In SAT.
  • Toybox 2018. Toybox Website. http://landley.net/toybox/. Toybox 2018. Toybox Website. http://landley.net/toybox/.
  • G.S. Tseitin. 1983. On the Complexity of Derivation in Propositional Calculus. In Automation of Reasoning J. Wrightson et al. (Eds.). G.S. Tseitin. 1983. On the Complexity of Derivation in Propositional Calculus. In Automation of Reasoning J. Wrightson et al. (Eds.).
  • S. Vasishth and M. Broe. 2011. The Foundations of Statistics: A Simulation-based Approach. Springer Berlin Heidelberg. S. Vasishth and M. Broe. 2011. The Foundations of Statistics: A Simulation-based Approach. Springer Berlin Heidelberg.
  • M. Velez P. Jamshidi N. Siegmund S. Apel and C. Kästner. 2021. White-Box Analysis Over Machine Learning: Modeling Performance of Configurable Systems. In ICSE. M. Velez P. Jamshidi N. Siegmund S. Apel and C. Kästner. 2021. White-Box Analysis Over Machine Learning: Modeling Performance of Configurable Systems. In ICSE.
  • VP9 Website 2021. VP9 Website. https://www.webmproject.org/vp9/. VP9 Website 2021. VP9 Website. https://www.webmproject.org/vp9/.
  • J. White B. Dougherty and D.C. Schmidt. 2009. Selecting Highly Optimal Architectural Feature Sets with Filtered Cartesian Flattening. Journal of Systems and Software(Aug. 2009). J. White B. Dougherty and D.C. Schmidt. 2009. Selecting Highly Optimal Architectural Feature Sets with Filtered Cartesian Flattening. Journal of Systems and Software(Aug. 2009).
  • J. White B. Doughtery and D. Schmidt. 2008. Filtered Cartesian Flattening: An Approximation Technique for Optimally Selecting Features While Adhering to Resource Constraints. In SPLC. J. White B. Doughtery and D. Schmidt. 2008. Filtered Cartesian Flattening: An Approximation Technique for Optimally Selecting Features While Adhering to Resource Constraints. In SPLC.
  • Wikipedia. 2016. 68-95-99.7 Rule. https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule. Wikipedia. 2016. 68-95-99.7 Rule. https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule.
  • Wikipedia. 2018. Conjunctive Normal Form. https://en.wikipedia.org/wiki/Conjunctive_normal_form. Wikipedia. 2018. Conjunctive Normal Form. https://en.wikipedia.org/wiki/Conjunctive_normal_form.
  • Wikipedia. 2021. Box Plot. https://en.wikipedia.org/wiki/Box_plot. Wikipedia. 2021. Box Plot. https://en.wikipedia.org/wiki/Box_plot.
  • Wikipedia. 2021. Integer Programming. https://en.wikipedia.org/wiki/Integer_programming. Wikipedia. 2021. Integer Programming. https://en.wikipedia.org/wiki/Integer_programming.
  • Wikipedia. 2021. Random Search. https://en.wikipedia.org/wiki/Random_search. Wikipedia. 2021. Random Search. https://en.wikipedia.org/wiki/Random_search.
  • Wikipedia. 2022. Chain Rule. https://en.wikipedia.org/wiki/Chain_rule. Wikipedia. 2022. Chain Rule. https://en.wikipedia.org/wiki/Chain_rule.
  • Wikipedia. 2022. Equisatisfiability. https://en.wikipedia.org/wiki/Equisatisfiability. Wikipedia. 2022. Equisatisfiability. https://en.wikipedia.org/wiki/Equisatisfiability.
  • Wikipedia. 2022. Order Statistics. https://en.wikipedia.org/wiki/Order_statistic. Wikipedia. 2022. Order Statistics. https://en.wikipedia.org/wiki/Order_statistic.
  • Y. Xue and Y. Li. 2018. Multi-Objective Integer Programming Approaches for Solving Optimal Feature Selection Problem. In ICSE. Y. Xue and Y. Li. 2018. Multi-Objective Integer Programming Approaches for Solving Optimal Feature Selection Problem. In ICSE.
  • T. Ye and S. Kalyanaraman. 2003. A Recursive Random Search Algorithm for Large-Scale Network Parameter Configuration. PER (June 2003). T. Ye and S. Kalyanaraman. 2003. A Recursive Random Search Algorithm for Large-Scale Network Parameter Configuration. PER (June 2003).
  • Y. Zhang J. Guo E. Blais and K. Czarnecki. 2015. Performance Prediction of Configurable Software Systems by Fourier Learning. In ASE. Y. Zhang J. Guo E. Blais and K. Czarnecki. 2015. Performance Prediction of Configurable Software Systems by Fourier Learning. In ASE.
  • Y. Zhu et al. 2017. BestConfig: Tapping the Performance Potential of Systems Via Automatic Configuration Tuning. In SoCC. Y. Zhu et al. 2017. BestConfig: Tapping the Performance Potential of Systems Via Automatic Configuration Tuning. In SoCC.
  • M. Zuluaga G. Sergent A. Krause and M. Püschel. 2013. Active Learning for Multi-Objective Optimization. In PMLR. M. Zuluaga G. Sergent A. Krause and M. Püschel. 2013. Active Learning for Multi-Objective Optimization. In PMLR.